Research › Analysis

Gradients Are Not All You Need

Metz, L. et al. · arXiv:2111.05803 (2021) · Paper

Q2 Computing analysis. All mathematical results and empirical findings are attributed to the original authors. We present our reading of the work as it relates to robotic automation.

What this paper solves

Differentiable programming has enabled gradient-based optimization through physics simulations, rendering pipelines, and recurrent models. The implicit assumption is that gradients computed through these systems are informative about the optimization landscape. This paper systematically characterizes the conditions under which this assumption fails: chaotic dynamical systems, long unrolled computation graphs, and discontinuous or non-smooth loss surfaces all produce gradients that are either numerically unstable or point in the wrong direction.

The practical consequence for robotics is that differentiable physics training is not universally applicable. When the simulation is chaotic or the training horizon is long, gradient estimates through the simulation are unreliable and gradient-free alternatives may perform better.

Key mathematical framework

Gradient explosion in chaotic systems

Consider a discrete dynamical system $x_{t + 1} = f (x_{t}, θ)$ . The gradient of the loss at time $T$ with respect to the initial parameter $θ$ is computed by the chain rule:

\frac{\partial L _{T}}{\partial θ} = \frac{\partial L _{T}}{\partial x _{T}} t = 1 \prod T \frac{\partial f ( x _{t} , θ )}{\partial x _{t - 1}} \cdot \frac{\partial f ( x _{1} , θ )}{\partial θ}

For a chaotic system, the Jacobian product $\prod_{t = 1}^{T} \partial f / \partial x_{t - 1}$ has eigenvalues that grow exponentially in the Lyapunov exponent $λ > 0$ :

t = 1 \prod T \frac{\partial f}{\partial x _{t - 1}} \sim e^{λ T}

Even small differences in $θ$ lead to exponentially diverging trajectories, making the gradient both large and poorly conditioned as a search direction. Gradient clipping addresses the magnitude but not the direction.

Bias from truncated gradients

Truncated backpropagation through time (TBPTT) limits the unroll horizon to $K$ steps to control compute and gradient explosion. The truncated gradient is:

\overset{g}{^}_{K} = \frac{\partial}{\partial θ} t = T - K \sum T ℓ_{t}

This is a biased estimator of the full gradient $g = \partial / \partial θ \sum_{t = 1}^{T} ℓ_{t}$ because it ignores the contribution of $θ$ through timesteps $1, \dots, T - K - 1$ . The bias is:

E [\overset{g}{^}_{K}] - g = - \frac{\partial}{\partial θ} t = 1 \sum T - K - 1 ℓ_{t}

which is nonzero whenever the loss at early timesteps depends on $θ$ . For long-horizon tasks, this term dominates and the truncated gradient points in the wrong direction.

Gradient variance from environment stochasticity

Stochastic environments contribute variance to the gradient estimator. Monte Carlo gradient estimation requires many samples to achieve low variance:

Var [\overset{g}{^}_{K}] = O (\frac{e ^{2 λ K}}{N})

where $N$ is the number of rollouts. Exponential growth in $K$ means the sample count required for a reliable gradient estimate grows exponentially with unroll horizon, making long-horizon differentiable training impractical even with many parallel environments.

Empirical results

Chaotic loss surfaces: Gradient descent fails on simple logistic maps and pendulum systems where the Lyapunov exponent exceeds the gradient signal, while evolutionary strategies succeed
Horizon scaling: Gradient-based methods degrade systematically as unroll horizon increases; gradient-free methods are unaffected by horizon length
Discontinuity: Contact discontinuities in physics simulation produce zero gradient almost everywhere with infinite gradient at contact transitions: the gradient is uninformative except at a measure-zero set

What this means for robotic automation

This paper characterizes the failure regime of differentiable physics training. The conditions that cause failure are common in robotics: contact dynamics are discontinuous, fluid dynamics are chaotic, and multi-step rollouts through any complex environment accumulate gradient bias. The practical implication is that differentiable physics is reliable for short-horizon, smooth, non-chaotic dynamics, and unreliable otherwise.

For swarm coordination with many interacting agents, the dynamics can exhibit emergent chaos: small per-agent perturbations can amplify through the interaction network. Gradient-based policy optimization in this setting faces exactly the failure modes characterized here. The appropriate response is either to shorten the horizon over which gradients are computed (accepting bias), or to use a gradient-free training algorithm (PPO, CMA-ES) for the inter-agent coordination component while reserving differentiable training for the per-agent dynamics where the assumptions hold.