Research › Analysis

Proximal Policy Optimization Algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. · arXiv:1707.06347 (2017) · Paper

Q2 Computing analysis. All mathematical results and empirical findings are attributed to the original authors. We present our reading of the work as it relates to robotic automation.

What this paper solves

Policy gradient methods for reinforcement learning are unstable: a single large gradient step can collapse a policy, and recovering requires restarting from scratch. Trust Region Policy Optimization (TRPO) addressed this by constraining each update to stay within a KL-divergence bound from the current policy, but it requires second-order optimization and is difficult to apply when the policy shares parameters with a value function or auxiliary loss.

PPO achieves the same stability guarantee through a first-order clipped surrogate objective that is simple to implement, compatible with shared architectures, and admits multiple minibatch epochs per rollout. It became the standard policy gradient algorithm for continuous control and is the training method used in Champion-Level Drone Racing (#11 in this archive).

Key mathematical framework

Policy gradient baseline

Standard policy gradient maximizes the expected return by ascending the gradient of the log-probability of actions weighted by advantage:

L^{P G} (θ) = E_{t} [lo g π_{θ} (a_{t} ∣ s_{t}) A_{t}]

where $A_{t}$ is an estimate of the advantage function at timestep $t$ . Multiple epochs of gradient ascent on this objective destroys the policy because large updates shift $π_{θ}$ far from the distribution that generated the data.

Probability ratio

Let $r_{t} (θ)$ be the ratio of the new policy probability to the old policy probability for the taken action:

r_{t} (θ) = \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{θ_{old}} ( a _{t} ∣ s _{t} )}

When $r_{t} = 1$ , the policy is unchanged. TRPO constrains updates by requiring the expected KL divergence between old and new policy to stay below a threshold $δ$ . PPO enforces a softer version by clipping $r_{t}$ directly.

Clipped surrogate objective

The PPO-Clip objective pessimistically clips the probability ratio so that updates which move the policy too far from the old policy receive no additional gradient signal:

L^{C L I P} (θ) = E_{t} [min (r_{t} (θ) A_{t}, clip (r_{t} (θ), 1 - ε, 1 + ε) A_{t})]

The clip threshold $ε = 0.2$ is the primary hyperparameter. When the advantage is positive (the action was good), the ratio is clipped above at $1 + ε$ so the policy cannot over-exploit it. When the advantage is negative (the action was bad), the ratio is clipped below at $1 - ε$ so the policy cannot collapse toward avoiding it too aggressively.

Generalized Advantage Estimation

The advantage estimate $A_{t}$ uses Generalized Advantage Estimation (GAE) to balance bias and variance. Let $δ_{t}^{V} = r_{t} + γ V (s_{t + 1}) - V (s_{t})$ be the TD residual. GAE computes an exponentially weighted sum of TD residuals:

\hat{A}_{t}^{G A E (γ, λ)} = l = 0 \sum \infty (γ λ)^{l} δ_{t + l}^{V}

The parameter $λ \in [0, 1]$ interpolates between high-bias one-step TD ( $λ = 0$ ) and high-variance Monte Carlo returns ( $λ = 1$ ). Typical values are $γ = 0.99, λ = 0.95$ .

Combined training objective

When the policy and value function share network parameters, PPO optimizes a combined objective that adds a value function loss and an entropy bonus:

L^{C L I P + V F + S} (θ) = E_{t} [L_{t}^{C L I P} (θ) - c_{1} L_{t}^{V F} (θ) + c_{2} S [π_{θ}] (s_{t})]

L_{t}^{V F} (θ) = (V_{θ} (s_{t}) - V_{t}^{targ})^{2}

The entropy bonus $S [π_{θ}] (s_{t})$ penalizes premature convergence to deterministic policies. Coefficients $c_{1} = 1, c_{2} = 0.01$ are standard. Unlike TRPO, this objective is differentiable and can be optimized with any first-order method.

Empirical results

Continuous control: PPO outperforms TRPO, A2C, and CMA-ES on the majority of MuJoCo locomotion benchmarks including Hopper, Walker2d, HalfCheetah, and Ant
Atari games: Matches or exceeds A3C performance on most Atari 2600 games using a single GPU within hours of wall-clock time
Sample efficiency: Achieves competitive performance with far fewer total environment interactions than TRPO due to multiple epochs of minibatch updates per rollout
Implementation simplicity: Requires only first-order gradients and no line search, making it compatible with standard deep learning frameworks and shared policy-value architectures

What this means for robotic automation

PPO's importance in this archive is primarily as an infrastructure algorithm. It is the training method behind Champion-Level Drone Racing, where 100 parallel simulation agents trained a quadrotor policy over 10⁸ environment interactions to achieve human-world-champion performance. The clip mechanism is what makes that scale of parallel training stable: independent rollouts from different agents can be batched into minibatch updates without any single rollout catastrophically shifting the policy.

The GAE formulation is directly applicable to any control problem where the reward signal is delayed or sparse. In swarm coordination, individual node rewards depend on collective outcomes that may not materialize until many timesteps after the action that caused them. GAE's weighted sum of TD residuals provides a low-variance advantage estimate under exactly this kind of temporal credit assignment problem.