TartanAir: A Dataset to Push the Limits of Visual SLAM

What this paper solves

Benchmarking visual SLAM and odometry systems requires diverse environments with accurate ground-truth trajectories. Existing datasets (EuRoC, KITTI, TUM) cover a narrow range of conditions and provide trajectories under controlled, well-lit, feature-rich settings. Systems trained and evaluated exclusively on these datasets fail to generalize to degraded conditions: rain, fog, darkness, featureless surfaces, and rapid lighting changes.

TartanAir addresses this by generating large-scale, photorealistic simulation data across diverse environments and degradation conditions with precise ground-truth from the simulation engine itself. It provides the training and evaluation substrate that validates zero-shot generalization from simulation to physical deployment.

Key mathematical framework

Photometric consistency and depth

TartanAir provides stereo RGB, depth maps, optical flow, and semantic segmentation from an Unreal Engine simulation. The depth map provides a direct measurement of scene geometry:

where $f$ is focal length and $B$ is stereo baseline. Exact ground-truth depth eliminates the ambiguity present in real-world LiDAR scans (occlusion, specular reflection, sparse coverage) and provides a clean signal for training depth-based navigation networks.

Trajectory ground truth

Ground-truth trajectory is expressed as a sequence of rigid-body poses:

The simulation engine provides $T_k$ with numerical precision limited only by floating-point arithmetic, compared to real-world datasets where ground truth is provided by motion capture or RTK-GPS with millimeter-level noise. Trajectory evaluation computes absolute trajectory error (ATE):

and relative pose error (RPE) over segments of length $\Delta$:

Domain diversity

TartanAir spans 16 environments across five categories: outdoor, urban, indoor, nature, and industrial. Each environment includes 22 degradation conditions combining lighting variation (day/night/dawn), weather (fog/rain/snow), and camera perturbations. For a learned system, the diversity ensures that the hypothesis space explored during training covers the modes likely to appear during physical deployment.

Empirical results

• Scale: Over 1 million frames across 16 environments, 22 degradation conditions, stereo RGB + depth + optical flow + semantic labels
• Baseline evaluation: Most tested visual SLAM systems (ORB-SLAM2, DSO, OpenVINS) fail in degraded conditions (fog, darkness, featureless surfaces) that TartanAir includes by design
• Challenge winner: The TartanAir SLAM challenge revealed that learned odometry systems generalize better across conditions than classical feature-based systems
• Ground truth precision: Sub-millimeter pose accuracy from simulation engine, unreachable by any real-world ground-truth system

What this means for robotic automation

TartanAir validates the simulation-first training paradigm by demonstrating that diverse simulated data transfers to real-world performance better than small real-world datasets. The key variable is diversity of conditions, not photorealism: a system trained on 22 degradation modes generalizes better than one trained on perfect synthetic imagery of a single environment type.

For zero-shot sim-to-real transfer, the relevant property is that the distribution of training conditions covers the distribution of test conditions. TartanAir's design makes this tractable by providing ground-truth data under conditions (severe fog, total darkness, texture-free surfaces) that are nearly impossible to annotate in the real world. Systems trained and validated on TartanAir carry an implicit certificate that their performance envelope spans these modes.