Source: Hugging Face Deep RL Course, Unit 8 — notes from Introduction through Additional Readings, reorganized for review. Continues from Unit 7.
Keypoints:
Unit 6 covered Advantage Actor-Critic (A2C), a hybrid of value-based and policy-based methods that stabilizes training by reducing variance with:
This unit covers Proximal Policy Optimization (PPO) — an architecture that improves the agent’s training stability by avoiding policy updates that are too large. It does this with a ratio indicating the difference between the current and old policy, and clips that ratio to the range $[1-\epsilon,\, 1+\epsilon]$.
The unit is in two parts:
The idea: improve training stability by limiting how much the policy changes at each training epoch — i.e. avoid too large of a policy update.
Two reasons:
The course’s picture is a hiker on a ridge: taking huge strides toward “higher reward” can send you over the edge, whereas short, cautious strides climb reliably.
Key insight: the collapse is not just “a bad update.” A bad policy collects bad data, and since a policy-gradient method learns only from the data its own policy generates, a bad policy can be self-reinforcing — there is no replay buffer of good experience to fall back on.
So with PPO we update the policy conservatively. To do that we need to measure how much the current policy has changed relative to the former one — via a ratio between current and former policy — and then clip that ratio into $[1-\epsilon, 1+\epsilon]$, which removes the incentive for the current policy to go too far from the old one (hence proximal policy optimization).
In REINFORCE the objective was
\[L^{PG}(\theta) = \hat{\mathbb{E}}_t\Big[\, \log \pi_\theta(a_t \mid s_t)\, \hat{A}_t \,\Big]\]Taking a gradient ascent step on this (equivalently gradient descent on its negative) pushes the agent to take actions leading to higher rewards and avoid harmful actions.
But the problem is the step size:
| Step size | Consequence |
|---|---|
| Too small | the training process is too slow |
| Too high | too much variability in training — possible collapse |
And the step size in parameter space is not the same thing as the step size in policy space: a small change in $\theta$ can produce a huge change in $\pi_\theta$. That’s exactly what we want to control.
PPO’s answer is a new objective, the Clipped Surrogate Objective function, which constrains the policy change to a small range using a clip and is designed to avoid destructively large weight updates.
It is the probability of taking action $a_t$ at state $s_t$ under the current policy, divided by the same probability under the previous policy. So $r_t(\theta)$ denotes the probability ratio between the current and old policy:
| Value of $r_t(\theta)$ | Meaning |
|---|---|
| $r_t(\theta) > 1$ | the action $a_t$ at state $s_t$ is more likely under the current policy than the old one |
| $0 < r_t(\theta) < 1$ | the action is less likely under the current policy than the old one |
| $r_t(\theta) = 1$ | the two policies agree on this action — no change |
So the probability ratio is an easy way to estimate the divergence between old and current policy. Note $r_t(\theta_{old}) = 1$ by construction: at the start of every update pass, before any gradient step, the ratio is exactly 1 everywhere.
Key insight: the ratio is the importance-sampling weight. It’s what lets us reuse a batch collected under $\pi_{\theta_{old}}$ to estimate the gradient of $\pi_\theta$ — which is why PPO can do several epochs on the same data. It’s also the cheap stand-in for an explicit KL divergence: no second-order optimization, just a division.
The ratio can replace the log-probability used in the policy objective, giving the left half of PPO’s new objective — the ratio times the advantage:
\[L^{CPI}(\theta) = \hat{\mathbb{E}}_t\Big[\, r_t(\theta)\, \hat{A}_t \,\Big] = \hat{\mathbb{E}}_t\!\left[\, \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)}\, \hat{A}_t \,\right]\](from Proximal Policy Optimization Algorithms, Schulman et al.)
However, without a constraint, if an action is much more probable under the current policy than under the former one, this leads to a significant policy-gradient step and therefore an excessive policy update. Maximizing $r_t \hat{A}_t$ for a positive advantage is unbounded: the optimizer would happily drive $r_t \to \infty$, pushing that action’s probability toward 1 based on a single noisy advantage estimate.
So we need to penalize changes that lead to a ratio far away from 1. Two ways to do that:
| TRPO (Trust Region Policy Optimization) | PPO | |
|---|---|---|
| Mechanism | KL-divergence constraint placed outside the objective function | Clips the probability ratio directly inside the objective |
| Enforcement | A hard trust region, solved with a constrained (second-order / conjugate-gradient) optimization | A soft, first-order penalty — plain SGD/Adam |
| Cost | Complicated to implement and takes more computation time | Cheap; a few extra lines of code |
| Divergence measure | KL between old and new policy | The ratio $r_t(\theta)$ as a proxy |
| Practical use | Rare outside research | The default policy-gradient method |
By clipping the ratio, we ensure we do not have a too-large policy update, because the current policy can’t be too different from the old one.
Two pieces:
clip — the clipped ratio is $r_t(\theta)$ forced into the interval $[1-\epsilon,\, 1+\epsilon]$: anything below $1-\epsilon$ becomes $1-\epsilon$, anything above $1+\epsilon$ becomes $1+\epsilon$. $\epsilon$ is the hyperparameter defining the clip range — in the paper $\epsilon = 0.2$, so the ratio can only vary from 0.8 to 1.2.min — we take the minimum of the clipped and non-clipped objective, so the final objective is a lower bound (a pessimistic bound) of the unclipped objective. Taking the minimum means we select either the clipped or the non-clipped objective depending on the ratio-and-advantage situation.⚠️ Do not confuse the two operations.
clipalone would not work. Clipping only makes the objective flat outside the interval; theminis what decides when that flatness applies. Without themin, a ratio pushed far past $1+\epsilon$ with a negative advantage would sit on a flat line and never be pulled back.
This is the crux of the unit. There are six situations, from the crossing of two binary/ternary choices: the sign of the advantage $\hat{A}_t$ (positive or negative) × where the ratio sits (below, inside, or above the interval). Remember throughout: we take the minimum of the clipped and unclipped objectives.
(The course reproduces the table from Daniel Bick’s thesis; below is the same content typeset.)
| # | $\hat{A}_t$ | Ratio $r_t(\theta)$ | Which term is the min? | Gradient | What happens |
|---|---|---|---|---|---|
| 1 | $> 0$ (good action) | inside $[1-\epsilon,\, 1+\epsilon]$ | unclipped | non-zero | Increase $\pi(a_t \mid s_t)$ — clipping does not apply |
| 2 | $< 0$ (bad action) | inside $[1-\epsilon,\, 1+\epsilon]$ | unclipped | non-zero | Decrease $\pi(a_t \mid s_t)$ — clipping does not apply |
| 3 | $> 0$ | below $1-\epsilon$ | unclipped | non-zero | We moved too far down on a good action ⇒ increase the probability (moving back toward the range) |
| 4 | $< 0$ | below $1-\epsilon$ | clipped | = 0 | Already much less likely than the old policy, and the action is bad — we don’t want to decrease further. Flat line ⇒ no weight update |
| 5 | $> 0$ | above $1+\epsilon$ | clipped | = 0 | Already much more likely than the old policy — don’t get too greedy. Flat line ⇒ no weight update |
| 6 | $< 0$ | above $1+\epsilon$ | unclipped | non-zero | We moved too far up on a bad action ⇒ decrease the probability (moving back toward the range) |
Clipping does not apply, since $r_t(\theta) \in [1-\epsilon, 1+\epsilon]$.
This is the “business as usual” regime: PPO behaves like an ordinary advantage-weighted policy gradient.
If $r_t(\theta) < 1-\epsilon$, the probability of taking that action at that state is much lower than under the old policy.
If $r_t(\theta) > 1+\epsilon$, the probability of taking that action at that state under the current policy is much higher than under the former policy.
We only update the policy with the unclipped objective part. When the minimum picks the clipped part, we don’t update the policy weights, because the gradient will equal 0.
So we update our policy only if:
Because when the ratio is clipped, the derivative is not the derivative of $r_t(\theta)\hat{A}_t$, but the derivative of either $(1-\epsilon)\hat{A}_t$ or $(1+\epsilon)\hat{A}_t$ — both of which are 0 with respect to $\theta$, since neither expression contains $\theta$ anymore. The clip has cut the gradient path.
| Direction of the move | What the objective does |
|---|---|
| Too far in the “good” direction (case 5: $\hat{A}_t > 0$, $r_t > 1+\epsilon$) | No further reward. Gradient 0 — the objective refuses to pay you for over-committing. We are not pulled back either; we simply stop. |
| Too far in the “bad” direction (case 6: $\hat{A}_t < 0$, $r_t > 1+\epsilon$) | Still penalized. Gradient non-zero — we are pushed back toward the range. |
| Too far down on a good action (case 3) | Still pushed back up toward the range. |
| Too far down on a bad action (case 4) | No further reward for going even lower. Gradient 0. |
The min is what creates this asymmetry: it makes the objective a pessimistic lower bound, so exceeding the trust region never buys you anything, while corrections that bring you back inside are always permitted. To summarize: thanks to the clipped surrogate objective, we restrict the range that the current policy can vary from the old one, because we remove the incentive for the ratio to move outside the interval — the clip forces the gradient to zero out there.
The full loss PPO actually optimizes is a combination of the clipped surrogate objective, a value loss, and an entropy bonus:
\[L_t(\theta) = \hat{\mathbb{E}}_t\Big[\, L^{CLIP}_t(\theta) \;-\; c_1\, L^{VF}_t(\theta) \;+\; c_2\, S[\pi_\theta](s_t) \,\Big]\]| Term | What it is | Why it’s there |
|---|---|---|
| $L^{CLIP}_t(\theta)$ | The clipped surrogate policy objective from §5 | Improves the policy, conservatively (the Actor) |
| $-\,c_1 L^{VF}_t(\theta)$ | The value-function loss, a squared error $L^{VF}t = (V\theta(s_t) - V_t^{\text{targ}})^2$; subtracted because we maximize $L_t$ but want to minimize this error | Trains the Critic, so the advantage estimates get better |
| $+\,c_2 S\pi_\theta$ | The entropy bonus — the entropy of the policy’s action distribution | Encourages exploration: rewards keeping the distribution spread out, delaying premature collapse to a deterministic policy |
$c_1$ and $c_2$ are coefficients (in the CleanRL defaults, vf-coef = 0.5 and ent-coef = 0.01).
Key insight: the sign convention matters and is a classic source of bugs. In code you minimize
pg_loss + c1 * v_loss - c2 * entropy, which is exactly $-L_t(\theta)$. Also note the actor and critic here typically share nothing but the optimizer (or share an encoder), which is why the coefficients are needed to balance the two gradients.
Putting it together, one PPO iteration looks like:
and the value targets are $\text{returns}_t = \hat{A}_t + V(s_t)$. The $\lambda$ knob (default 0.95) trades bias against variance: $\lambda = 0$ gives one-step TD, $\lambda = 1$ gives Monte-Carlo.
Key insight: step 4 is why PPO exists. Vanilla policy gradient must throw away each batch after a single gradient step, because the data is only valid for the policy that generated it. The ratio + clip let PPO squeeze $K$ epochs out of one batch while keeping the policy provably close to the one that collected the data — much better sample efficiency while remaining essentially on-policy. As soon as the ratio drifts past $1\pm\epsilon$, the clip switches those samples off.
In the CleanRL code this is literally:
ratio = (newlogprob - b_logprobs[mb_inds]).exp()
pg_loss1 = -mb_advantages * ratio
pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef)
pg_loss = torch.max(pg_loss1, pg_loss2).mean() # max of negatives == min of the objective
Note torch.max of the two negated terms is exactly the min of the objective.
The best way to understand an architecture is to implement it from scratch — done already for a value-based method (Q-Learning) and a policy-based one (REINFORCE).
Resources the course uses:
Environment: LunarLander-v2 — the very first environment in the course. Finally, push the trained model to the Hub with a replay video and an evaluation score.
| Argument | Default | Meaning |
|---|---|---|
--learning-rate | 2.5e-4 | Adam LR (with eps=1e-5) |
--anneal-lr | True | Linearly decay the LR to 0 over training |
--num-envs | 4 | Parallel (vectorized) environments |
--num-steps | 128 | Rollout length per env ⇒ batch size $= 4 \times 128 = 512$ |
--gamma | 0.99 | Discount factor |
--gae / --gae-lambda | True / 0.95 | Use GAE, with $\lambda = 0.95$ |
--num-minibatches | 4 | Minibatch size $=$ batch size $/\,4$ |
--update-epochs | 4 | $K = 4$ epochs over each collected batch |
--norm-adv | True | Normalize advantages per minibatch |
--clip-coef | 0.2 | This is $\epsilon$ |
--clip-vloss | True | Also clip the value loss, as in the paper |
--ent-coef | 0.01 | $c_2$, the entropy bonus coefficient |
--vf-coef | 0.5 | $c_1$, the value-loss coefficient |
--max-grad-norm | 0.5 | Global gradient-norm clipping |
--target-kl | None | Optional early stop of the epoch loop if approx-KL exceeds it |
Diagnostics the implementation logs that are worth knowing: approx_kl (an estimate $\mathbb{E}[(r-1) - \log r]$, from Schulman’s KL-approximation note) and clipfrac (fraction of samples with $\lvert r - 1 \rvert > \epsilon$). A clipfrac near 0 means the updates are timid; a very high one means the policy is straining against the trust region.
Sample Factory is one of the fastest RL libraries, focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO). It reaches SOTA performance in a variety of domains while minimizing training time and hardware requirements.
How it works: it spawns multiple processes running rollout workers, inference workers, and a learner worker. The workers communicate through shared memory, which lowers inter-process communication cost.
⚠️ Because the inference workers use a slightly stale copy of the policy, the data arriving at the learner is off-policy by a few updates. This is exactly the regime the clipped ratio is built to tolerate — asynchronous PPO is only sane because of the trust region.
Actor-Critic models in Sample Factory have three components:
| Component | Role |
|---|---|
| Encoder | Processes input observations (images, vectors) into a vector — the part you’d most likely customize |
| Core | Integrates vectors from one or more encoders; can optionally include a single/multi-layer LSTM/GRU for a memory-based agent |
| Decoder | Extra layers on the core’s output before computing the policy and value heads |
Other key features: serial (single-process) mode for easy debugging; CPU-based and GPU-accelerated environments; single- and multi-agent training, self-play, multiple policies at once on one or many GPUs; Population-Based Training (PBT); discrete, continuous and hybrid action spaces; vector, image and dictionary observation spaces; automatic model construction by parsing the action/observation spaces; WandB/TensorBoard summaries; and Hugging Face Hub integration.
ViZDoom is an open-source Python interface to the Doom engine, created in 2016 by Marek Wydmuch and Michał Kempka at Poznań University of Technology. It allows training agents directly from screen pixels across many scenarios (up to team deathmatch). Being a 90s game engine, it runs at accelerated speeds on modern hardware, so complex behaviors can be learned fairly quickly.
The scenario trained in the notebook: doom_health_gathering_supreme. The objective is to teach the agent to survive without knowing what makes it survive. The agent knows only that life is precious and death is bad, so it must learn what prolongs its existence and that its health is connected to survival. The map is a rectangle with walls and a green, acidic floor that periodically hurts the player; medkits are spread over the map and new ones fall from the sky. Medkits heal part of the player’s health, so to survive the agent must pick them up. The episode ends on death or timeout. Certification target: mean reward $\geq 5$.
Notable run configuration: --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000. The optional harder challenge is doom_deathmatch_bots, which takes many hours on a machine beefier than Colab — the course instead has you download a pretrained checkpoint (edbeeching/doom_deathmatch_bots_2222) and watch it play.
The Part-2 hands-on was written by Edward Beeching, ML Research Scientist at Hugging Face (also behind Godot RL Agents).
| REINFORCE | A2C | PPO | |
|---|---|---|---|
| Family | Pure policy gradient (Monte Carlo) | Actor-Critic (hybrid) | Actor-Critic (hybrid) |
| Objective weight on $\log \pi$ | Return $G_t$ | Advantage $A_t = Q(s,a) - V(s)$ | Advantage $\hat{A}_t$ × the clipped ratio |
| Needs full episodes? | Yes | No (bootstraps with the Critic) | No (GAE over a fixed-length rollout) |
| Variance | Very high | Reduced by the Critic baseline | Reduced by the Critic + advantage normalization |
| Gradient steps per batch | 1 | 1 | $K$ epochs of minibatches |
| Guards against huge updates? | No | No | Yes — the clip |
| Sample efficiency | Lowest | Middle | Best of the three (data reused $K$ times) |
| On-policy? | Yes | Yes | Yes, but tolerates slightly stale data |
| TRPO | PPO | |
|---|---|---|
| Constraint form | $\hat{\mathbb{E}}t[\,D{KL}(\pi_{\theta_{old}} \Vert \pi_\theta)\,] \leq \delta$, a hard constraint outside the objective | $\min(\cdot,\, \text{clip}(\cdot))$, inside the objective |
| Optimization | Constrained / second-order (Fisher-vector products, conjugate gradient, line search) | Unconstrained first-order — Adam, as usual |
| Guarantee | A genuine monotonic-improvement bound | Heuristic, but empirically strong |
| Per-state vs per-batch | Constrains the average KL over the batch | Clips per-sample ratios |
| Implementation cost | Complicated, more computation time | A handful of lines |
| Verdict | Elegant theory, painful engineering | The practical default |
Mnemonic for the clip: “Never pay for going too far; always allow coming back.”
clip and min. What would break if you kept only clip?approx_kl and clipfrac tell you during training? What does a clipfrac of ~0 suggest?These are optional readings if you want to go deeper.
PPO Explained
PPO Implementation details
Importance Sampling
Libraries used in the hands-on
Next up: the bonus units and advanced topics — model-based RL, offline vs online RL and Decision Transformers (treating RL as sequence modeling), generalization in RL, curriculum learning, and RLHF (Reinforcement Learning from Human Feedback), where the PPO you just learned becomes the workhorse for aligning language models.