Deep Drone Acrobatics

What this paper solves

Acrobatic quadrotor maneuvers require extreme angular accelerations that push platforms to their physical limits and cause vision-based state estimation to fail through motion blur and feature tracking loss. Prior systems performing such maneuvers relied on external motion capture infrastructure. This paper is the first to demonstrate acrobatic flight at up to 3g using only onboard sensing and computation, with zero real-world fine-tuning after simulation training.

The central insight is that raw sensory input contains simulation-reality discrepancies that prevent transfer, but appropriate abstractions of those inputs (feature tracks from VIO, pre-integrated IMU) are invariant to the environmental factors that differ between simulation and the physical world. The paper proves this formally and validates it across Power Loop, Barrel Roll, Matty Flip, and combined sequences.

Key mathematical framework

Quadrotor dynamics

The privileged expert operates on a simplified quadrotor model where $p^{WB}$ is position, $v^{WB}$ velocity, $q^{WB}$ attitude quaternion, ${\omega }^B$ body angular rate, and $c^B$ collective thrust in body frame:

This model neglects angular rate dynamics to reduce the planning problem to a form tractable for real-time MPC. The simplification holds for the maneuver timescales tested.

Receding horizon expert controller

The privileged expert ${\pi }^{\ast }$ is a Model Predictive Controller that repeatedly solves an open-loop optimization over a horizon of $N$ steps. Let $x[k]={\tau }_r[k]-s[k]$ be the deviation of platform state from the reference trajectory at step $k$. The MPC minimizes:

subject to equality constraints $r(x,u)=0$ from the system dynamics and optional bounds $h(x,u)\le 0$. The cost matrices$Q,R$ are positive semidefinite. Only the first element of the optimized control sequence is applied.

Student-expert performance bound

The student policy $\pi$ imitates the privileged expert via DAgger. The performance gap is formally bounded by the Wasserstein distance between the two policies on the trajectories the student actually visits:

where $C_{{\pi }^{\ast }}$ depends on the smoothness of expert actions. This reduces the learning problem to minimizing the expected action discrepancy under the student's own state distribution:

Formal proof that abstraction reduces the sim-to-real gap

Let $M(z|s)$ and $L(z|s)$ be the observation models in the real world and in simulation. For a Lipschitz-continuous policy with constant $K$, the simulation-to-reality performance gap is bounded by the Wasserstein distance between observation models:

If a mapping $f$ satisfies $D_W(f(M),f(L))\le D_W(M,L)$, then a policy trained on the abstracted representation $f(O)$ has a strictly lower simulation-to-reality gap than one trained on raw observations. Feature tracks satisfy this condition: they depend on scene geometry rather than surface appearance, making them invariant to the visual domain shift between simulation and reality.

Network architecture

Three asynchronous input branches process different modalities at their native sensor frequencies and feed a synchronous output MLP:

• Visual branch (feature tracks): Reduced PointNet (32 → 64 → 128 → 128 filters) over 40 sampled Harris corners per keyframe, global average pooling to 128-dim, then temporal convolutions over history length $L=8$
• IMU branch: Bias-subtracted, gravity-aligned inertial signal at 100 Hz through temporal convolution (128 filters then 3 × 64 filters), fully connected to 128-dim
• Reference branch: Same structure as IMU branch, updated at 50 Hz
• Output MLP: Concatenated 384-dim input through layers of size 128 → 64 → 32, outputting collective thrust and body rates at 100 Hz

Empirical results

• Tracking error: 24 ± 5 cm on Power Loop, 58 ± 9 cm on Barrel Roll, 53 ± 15 cm on Matty Flip, 45% lower than VIO-MPC baseline across maneuvers
• Success rate: 100% on all three individual maneuvers in simulation and on the physical platform with zero fine-tuning
• Combo sequence: 95% success on a combined triple Barrel Roll + double Power Loop + Matty Flip executed without stopping
• Abstraction validation: Raw image input achieved 80% success in training environment and 0% in novel backgrounds; feature track input maintained 100% success across all tested environments
• Hardware: Custom 1.15 kg quadrotor with 4:1 thrust-to-weight ratio, NVIDIA Jetson TX2 for inference, Intel RealSense T265 for visual-inertial input
• Peak acceleration: Up to 3g during acrobatic maneuvers

What this means for robotic automation

The abstraction lemma is the transferable result. The paper provides a formal proof that using scene-geometry-dependent representations rather than raw pixel input bounds the sim-to-real gap in terms of the Wasserstein distance between observation models. This is not a heuristic: it is a theorem with a computable certificate. Any system training in simulation can apply the same reasoning to choose input representations that minimize domain shift.

The DAgger-based training procedure handles the distribution shift between expert trajectories and student trajectories without requiring physical hardware during training. The student's own rollouts are added to the dataset iteratively, ensuring the learned policy is robust on the state distribution it will actually visit at deployment rather than only on trajectories the expert generates.

The asynchronous network design (where each sensor branch operates at its native frequency) is directly applicable to any multi-sensor embedded system where sensors produce data at different rates. The output layer runs at a fixed frequency regardless of when inputs arrive, which is the correct architecture for real-time control on constrained hardware.