Research › Analysis

Differentiable Physics Simulation

Liang, J. & Lin, M. C. · ICLR Workshop on Integration of Deep Neural Models and Differential Equations (2020)

Q2 Computing analysis. All mathematical results and empirical findings are attributed to the original authors. We present our reading of the work as it relates to robotic automation.

What this paper solves

Training robot control policies from scratch with model-free reinforcement learning requires millions of environment interactions and often converges to locally optimal solutions. Differentiable physics simulation provides an alternative: by making the simulation itself differentiable, gradients of a task objective can be propagated directly through the physical dynamics to the policy parameters, dramatically reducing the sample count required to find good policies.

This work characterizes the theory and implementation of differentiable rigid-body simulation, including contact dynamics and joint constraints, and demonstrates that gradient-based optimization through physics can solve manipulation and locomotion tasks with orders of magnitude fewer interactions than model-free methods on the same benchmarks.

Key mathematical framework

Differentiable time integration

Rigid-body dynamics integrate Newton-Euler equations forward in time. For a rigid body with generalized coordinates $q \in R^{n}$ and velocities $\overset{q}{˙} \in R^{n}$ , the dynamics are:

M (q) \overset{q}{¨} + C (q, \overset{q}{˙}) \overset{q}{˙} = τ + J_{c}^{⊤} λ

where $M (q)$ is the mass matrix, $C (q, \overset{q}{˙})$ contains Coriolis and centrifugal terms, $τ$ is the generalized force from actuators, $J_{c}$ is the contact Jacobian, and $λ$ are contact forces. Standard simulators solve this forward for $\overset{q}{¨}$ at each timestep. A differentiable simulator additionally maintains the Jacobians:

\frac{\partial q _{t + 1}}{\partial θ}, \frac{\partial q ˙ _{t + 1}}{\partial θ}

where $θ$ are policy parameters, enabling backpropagation through the physical trajectory.

Gradient through the control policy

The policy $π_{θ} (x_{t})$ maps observed state to control. The task loss at the end of a trajectory of length $T$ is:

L (θ) = ℓ (q_{T}, \overset{q}{˙}_{T})

The policy gradient is computed by unrolling through the simulation:

\frac{\partial L}{\partial θ} = \frac{\partial ℓ}{\partial q _{T}} \frac{\partial q _{T}}{\partial θ} + \frac{\partial ℓ}{\partial q ˙ _{T}} \frac{\partial q ˙ _{T}}{\partial θ}

Each step of the unroll applies the chain rule through the time integrator, the contact resolution step, and the policy network. All three must be differentiable for the gradient to flow end-to-end.

Contact handling

Contact forces are the primary source of non-differentiability in rigid-body simulation. The paper addresses this through smooth approximations of the contact complementarity conditions. The Signorini condition (no penetration, no tensile force) is approximated by a smooth penalty:

λ_{n} \approx k max (0, - g_{n})^{2}

where $g_{n}$ is the signed gap between contacting surfaces and $k$ is a stiffness coefficient. This penalty is differentiable everywhere except at exact zero gap, which has measure zero in practice.

Empirical results

Sample efficiency: Differentiable physics optimization converges to good policies in 10-100x fewer environment interactions than PPO and SAC on manipulation benchmarks
Policy quality: Final performance matches or exceeds model-free baselines, with the gradient signal providing better local search direction than Monte Carlo estimates
Locomotion: Locomotion gaits (walk, trot, jump) emerge from optimization against a forward progress objective without reward shaping, using gradients through contact forces

What this means for robotic automation

Differentiable physics simulation is the enabling technology behind Back to Newton's Laws (#1). The ability to propagate gradients directly through the flight dynamics is what allows Back to Newton's Laws to train a policy end-to-end without a reward function, instead minimizing the discrepancy between predicted and actual velocity using gradients that flow through the aerodynamic model.

The contact differentiability problem is less relevant for aerial vehicles operating below 7 m/s in atmospheric conditions, where contact dynamics are absent and the relevant physics is aerodynamic drag. The smooth drag model in that regime means the differentiable physics assumption holds without any contact approximation. Differentiability breaks down for ground-contact robotics (landing, manipulation) but is clean for free-flight dynamics within the operating envelope.