Source: Hugging Face Deep RL Course, Bonus Unit 3 — notes from Introduction through Brief introduction to RL documentation, reorganized for review. Continues from Unit 8.
Keypoints:
You’ve finished the main course, so you have a solid background in Deep RL: Q-Learning → Deep Q-Learning → policy gradients / REINFORCE → A2C → PPO → multi-agent. But the course was just the beginning — there are many subfields left to discover.
This optional unit gives you resources to explore multiple concepts and research topics in RL. Contrary to other units, it’s a collective work of multiple people from Hugging Face, and the author is credited per section:
| Section | Author |
|---|---|
| Offline vs. Online RL, Environments to try | Thomas Simonini |
| Model-Based RL, RLHF, RL documentation | Nathan Lambert |
| Curriculum Learning, Language models in RL | Clément Romac |
| Decision Transformers, Godot RL Agents | Edward Beeching |
So: read this unit as an index of directions, and treat the “Further reading” links as the real content.
Every advanced topic below exists because some assumption of vanilla online deep RL breaks. That’s the cleanest way to memorize the unit:
| Topic | Assumption it attacks | Problem it addresses | Core idea in one line |
|---|---|---|---|
| Offline RL | “I can freely interact with the env” | Real-world interaction is expensive/unsafe; you already have logged data | Learn a policy purely from a fixed dataset, no environment interaction |
| Generalisation | “Train env == test env” | Agents overfit a single level/layout; the real world is non-stationary and open-ended | Train over variations (procedural generation, domain randomization); measure the generalisation gap |
| Model-based RL | “Model-free is enough” | Sample inefficiency — millions of steps per task | Learn a dynamics model and plan with it (e.g. MPC) |
| Curriculum Learning | “One fixed task, learn it” | Hard / sparse-reward tasks needing incremental skill acquisition; robustness to env variations | Order tasks easy→hard; ACL learns that ordering automatically |
| Decision Transformers | “RL = maximize a value function” | Bootstrapping/value fitting is unstable, and offline data is plentiful | Recast RL as conditional sequence modeling: condition on desired return, predict actions |
| LMs in RL | “Tabula rasa learning” | Learning from scratch ⇒ sample inefficiency + behaviors humans find bizarre | Use an LM’s world knowledge as prior; RL grounds/corrects it |
| RLHF | “A reward function exists and I can write it” | Human preferences are hard to specify as an equation | Learn a reward model from human comparisons, then RL fine-tune |
| Godot RL / Learning Agents | “Gym envs are enough” | You want agents in your own game | Bridge a game engine (Godot / Unreal) to Python RL frameworks |
| Reward Reports | “Ship it and forget it” | RL systems change over time once deployed with humans in the loop | Living documentation with a change-log + update triggers |
Deep RL agents learn from batches of experience. The whole distinction here is: how is that experience collected?
The agent gathers data directly: it collects a batch of experience by interacting with the environment, then uses that experience immediately (or via a replay buffer) to update its policy.
But this implies you either train the agent directly in the real world, or you have a simulator. If you don’t have one you must build it, which is:
The agent only uses data collected from other agents or human demonstrations. It does not interact with the environment at all. The process:
Why this is attractive: safety (no dangerous exploration in the real world), cost (no simulator to build, no robot to break), and data reuse — you can learn from logs you already have, the way supervised learning reuses static datasets.
⚠️ The counterfactual queries problem (the course’s name for it; in the literature this is distributional shift / out-of-distribution actions): what do we do if our agent decides to do something for which we don’t have data? For instance, it wants to turn right at an intersection, but no trajectory in the dataset ever turned right there.
Because there is no exploration, the learned $Q$ or policy can confidently extrapolate garbage on actions the dataset never took, and nothing in the pipeline can correct it — there is no environment to disprove the optimistic estimate. This is why offline RL algorithms add conservatism/constraint machinery (staying near the behavior policy, penalizing OOD actions) rather than just running DQN on a static buffer.
Solutions exist; the course points you to a video overview of offline RL.
| Online RL | Offline RL | |
|---|---|---|
| Data source | The agent’s own interaction | A fixed, pre-collected dataset (other agents / humans) |
| Env interaction during training | Yes | No |
| Needs simulator or real-world access | Yes | No |
| Can explore to fix its own mistakes | Yes | No |
| Main failure mode | Sample cost, unsafe exploration, sim exploitation | Counterfactual queries / distributional shift |
| Feels like | Trial and error | Supervised learning on trajectories |
Key insight: offline RL turns RL into something that looks like the rest of deep learning (a dataset + a loss), which is exactly why the Decision Transformer (§5) fits the offline setting so naturally.
Further reading (from the page):
(The course page names no dataset suite or library here — if you want to actually run offline RL, the standard benchmark in the wider literature is D4RL, and 🤗 Transformers ships the Decision Transformer, see §5.)
Generalisation plays a pivotal role in RL. RL algorithms demonstrate good performance in controlled environments, but the real world presents a unique challenge due to its non-stationary and open-ended nature.
So the goal for real-world RL is algorithms that are:
The terminology worth carrying away (the survey below is built around it):
| Term | Meaning |
|---|---|
| Overfitting in RL | The agent memorizes one environment instance / one level — including its exact layout, textures, seed — instead of learning the task |
| Procedural generation | Generate a distribution of levels programmatically, so train and test levels are different draws from the same distribution |
| Generalisation gap | Performance on training levels minus performance on held-out test levels — the RL analogue of train/test error gap |
| Domain randomization | Apply random variations to environment parameters (physics, visuals, dynamics) during training so the policy is invariant to them — the classic sim-to-real trick |
| Zero-shot vs few-shot transfer | Deploy on a new-but-related environment with no adaptation, vs with a small amount of fine-tuning |
Key insight: in supervised learning you’d never evaluate on the training set — but a huge amount of classic RL does exactly that (train and report on one fixed environment). Generalisation research is largely about fixing the evaluation protocol as much as the algorithm.
Further reading (from the page):
Note the natural link to §4: domain randomization is also a curriculum tool (OpenAI’s Rubik’s-cube hand), so generalisation and curriculum learning are two views of the same “train over a distribution of tasks” idea.
MBRL differs from its model-free counterpart only in learning a dynamics model — but that has substantial downstream effects on how decisions are made.
The dynamics model usually models the environment transition dynamics
\[s_{t+1} = f_\theta(s_t, a_t)\]but the same framework accommodates inverse dynamics models (mapping states to actions) or reward models (predicting rewards).
The agent acts in an MDP governed by a transition function $s_{t+1} = f(s_t, a_t)$ that returns a reward $r(s_t,a_t)$ at each step. From a collected dataset
\[\mathcal{D} := \{\, s_i,\, a_i,\, s_{i+1},\, r_i \,\}\]the agent learns a model $s_{t+1} = f_\theta(s_t,a_t)$ to minimize the negative log-likelihood of the transitions:
\[\mathcal{L}(\theta) \;=\; -\sum_{i \in \mathcal{D}} \log p_\theta\!\left(s_{i+1} \mid s_i, a_i\right)\]Then control is done with sample-based model-predictive control (MPC) using the learned model: optimize the expected reward over a finite, recursively predicted horizon $\tau$, from a set of actions sampled from a uniform distribution $U(a)$:
\[a_{t:t+\tau}^{*} \;=\; \arg\max_{a_{t:t+\tau} \sim U(a)} \; \sum_{k=0}^{\tau-1} r\!\left(\hat{s}_{t+k},\, a_{t+k}\right), \qquad \hat{s}_{t+k+1} = f_\theta(\hat{s}_{t+k},\, a_{t+k})\]Only the first action of the winning sequence is executed; then you re-plan (that’s the “predictive control” part).
| Model-free (this course) | Model-based | |
|---|---|---|
| What is learned | Policy $\pi_\theta$ and/or value $Q_\theta$ | A dynamics model $f_\theta$ (plus optionally reward model) |
| How actions are chosen | Read them off the policy/value function | Plan by predicting the future (e.g. MPC) |
| Sample efficiency | Low — needs many env steps | High — the model amortizes experience |
| Compute at decision time | Cheap (one forward pass) | Expensive (rollouts / search per step) |
| Main failure mode | Sample cost, instability | Model error, which compounds over the horizon; the planner exploits model bugs |
⚠️ Compounding error is the defining MBRL problem: each predicted step feeds the next, so small one-step errors grow along $\tau$. Worse, the planner actively searches for the action sequence with the highest predicted reward — which is precisely where an optimistically-wrong model lives. Hence short horizons, model ensembles, and uncertainty penalties.
Further reading (from the page):
Most methods in this course work well in practice, but there are cases where using them alone fails — for instance when:
In such cases you need to propose different tasks to the agent and organize them so that skills are acquired progressively. This is Curriculum Learning, and it usually implies a hand-designed curriculum (a set of tasks in a specific order). In practice you can control:
With a sparse reward, a randomly-initialized policy in the hard task essentially never sees a reward, so the gradient carries no signal. An easy task variant is one where random behavior sometimes succeeds — that produces reward, which produces learning signal, which produces a policy good enough to occasionally succeed in the next-hardest variant. The curriculum is a ladder of non-zero-gradient tasks up to a task you could not have learned directly.
This is exactly the Curriculum Learning feature you met in Unit 5 with ML-Agents (e.g. progressively raising the wall height in WallJump / progressively enlarging the target): same idea, exposed as a config file where a lesson’s threshold on the reward triggers the next lesson.
Designing such a curriculum is not always trivial, so ACL proposes approaches that learn to create the organization of tasks in order to maximize the RL agent’s performance. Portelas et al. define ACL as:
… a family of mechanisms that automatically adapt the distribution of training data by learning to adjust the selection of learning situations to the capabilities of RL agents.
So ACL is a teacher whose action is “which task to serve next” and whose objective is the student’s learning progress — a nice second-order RL problem.
Concrete example: OpenAI used Domain Randomization (random variations applied to the environment) to make a robot hand solve Rubik’s Cubes — see OpenAI: Solving Rubik’s Cube with a Robot Hand.
You can also play with the robustness of trained agents in the TeachMyAgent benchmark by controlling environment variations — or even drawing the terrain yourself — in the interactive demo: Interactive DeepRL Demo (flowers-team).
Further reading (from the page):
Overview of the field
Recent methods
The Decision Transformer was introduced by Decision Transformer: Reinforcement Learning via Sequence Modeling, Chen L. et al.. It abstracts RL as a conditional-sequence-modeling problem.
Instead of training a policy with RL methods — such as fitting a value function that tells us which action maximizes the return — we use a sequence modeling algorithm (a Transformer) that, given a desired return, past states, and past actions, generates future actions to achieve that desired return.
It is an autoregressive model conditioned on the desired return, past states and past actions, generating future actions that achieve the desired return.
Concretely, the trajectory is fed as a flat token sequence of three modalities, with the return replaced by the return-to-go $\hat{R}_t$:
\[\tau \;=\; \big(\hat{R}_1,\, s_1,\, a_1,\; \hat{R}_2,\, s_2,\, a_2,\; \dots,\; \hat{R}_T,\, s_T,\, a_T \big), \qquad \hat{R}_t = \sum_{t'=t}^{T} r_{t'}\]and the model is trained with a plain supervised loss to predict $a_t$ from everything before it. At test time you specify the return you want (e.g. “get 6000”) as $\hat{R}_1$, feed the current state, read off the action, execute it, then decrement the return-to-go by the reward received, and repeat.
Key insight: this is a complete shift in the RL paradigm — we use generative trajectory modeling (modeling the joint distribution of the sequence of states, actions and rewards) to replace conventional RL algorithms. In Decision Transformers we don’t maximize the return; we generate a series of future actions that achieve the desired return.
The 🤗 Transformers team integrated the Decision Transformer — an Offline RL method — into the library and onto the Hugging Face Hub.
Learn / do:
Further reading (from the page):
Language models (LMs) show impressive abilities when manipulating text — question answering, even step-by-step reasoning. Additionally, training on massive text corpora let them encode various types of knowledge, including abstract knowledge about the physical rules of our world (what is possible to do with an object, what happens when you rotate an object, …).
The natural question: can this knowledge benefit agents such as robots solving everyday tasks? Early works showed interesting results, but the proposed agents lacked any learning method — which prevents them from adapting to the environment (e.g. fixing wrong knowledge) or learning new skills.
There’s a potential synergy: LMs bring knowledge about the world, RL aligns and corrects that knowledge by interacting with an environment.
This is especially interesting from the RL side, because RL mostly relies on the tabula rasa setup where everything is learned from scratch, leading to:
Two directions studied:
| Direction | Paper | What it does |
|---|---|---|
| Adapt / align the LM itself | Grounding Large Language Models with Online Reinforcement Learning | Aligns an LM to a textual environment using PPO. The knowledge in the LM gave fast adaptation to the environment (a path to sample-efficient RL agents) and better generalisation to new tasks once aligned |
| Keep the LM frozen, use it as a guide | Guiding Pretraining in Reinforcement Learning with Large Language Models | Leverages LM knowledge to guide the RL agent’s exploration toward human-meaningful and plausibly useful behaviors, without needing a human in the loop during training |
⚠️ Several limitations make this work still very preliminary: the agent’s observations must be converted to text before being given to the LM, and interacting with very large LMs is computationally expensive.
Further reading (from the page):
This is the section worth the most attention: it’s the bridge from this course to modern LLM training.
RLHF is a methodology for integrating human data labels into an RL-based optimization process.
Motivation: the challenge of modeling human preferences. For many questions, even if you could write down an equation for one ideal, humans differ on their preferences. Updating models based on measured data is an avenue to alleviate these inherently human ML problems — i.e. instead of hand-writing a reward function, learn it from data about what people actually prefer.
Notice this is the same failure mode as reward hacking in §1, approached from the other side: rather than fixing the simulator, we fix the reward.
Stage 1 — Pretraining / supervised fine-tuning. Start from a pretrained LM. Optionally supervised-fine-tune it on demonstration data of the desired behavior. Call the result the reference model $\pi_{\text{ref}}$; it is also the initialization of the policy $\pi_\theta$.
Stage 2 — Collect human preferences and train a reward model. Sample several completions $y_1, y_2, \dots$ for the same prompt $x$, and ask humans which one they prefer — comparisons, not absolute scores, because relative judgments are far more consistent between annotators. Train a reward model $r_\phi(x,y)$ (a scalar-head LM) so that the preferred (winning) completion $y_w$ scores above the rejected (losing) one $y_l$:
\[\mathcal{L}(\phi) \;=\; -\,\mathbb{E}_{(x,\,y_w,\,y_l)\sim \mathcal{D}}\Big[\log \sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)\Big]\]The reward model is thus a learned stand-in for the human, cheap enough to query millions of times inside an RL loop.
Stage 3 — RL fine-tuning (typically PPO). Treat the LM as a policy: the state is the prompt plus tokens generated so far, an action is the next token, and the reward is $r_\phi(x,y)$ at the end of the generation. Then run the PPO you learned in Unit 8 — with one crucial addition, a KL penalty that keeps the policy near the reference model:
\[\max_{\theta}\; \mathbb{E}_{x\sim\mathcal{D},\; y\sim\pi_\theta(\cdot\mid x)}\Big[\, r_\phi(x,y) \,\Big] \;-\; \beta\, \mathbb{D}_{\mathrm{KL}}\!\Big[\pi_\theta(y\mid x)\,\big\|\,\pi_{\text{ref}}(y\mid x)\Big]\]equivalently, RL is run on the shaped per-sample reward
\[r_{\text{total}}(x,y) \;=\; r_\phi(x,y) \;-\; \beta \log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\]Key insight — why the KL term is not optional. The reward model is only accurate on the distribution of text it was trained on. Unconstrained PPO will find high-scoring gibberish: it drifts off-distribution and over-optimizes the reward model rather than the humans it proxies. The KL penalty is a leash to $\pi_{\text{ref}}$, so the policy improves without leaving the region where $r_\phi$ is trustworthy. The scaling behavior of exactly this failure is studied in Scaling Laws for Reward Model Overoptimization.
Note the two distinct clipping/constraint mechanisms in play, which are easy to confuse:
| Mechanism | Between what | Purpose |
|---|---|---|
| PPO clipped objective / ratio clip | current policy vs the policy that collected this batch | Keep each update step small — optimization stability (Unit 8) |
| KL penalty $\beta$ | current policy vs the frozen reference model | Keep the final model near the pretrained LM — prevent reward-model over-optimization and loss of language quality |
2MBJOuVq380.)(copied from the Illustrating RLHF blog post; the field was popularized with the emergence of DeepRL around 2017 and grew into a broader study of LLM applications)
Pre-dating the LM focus:
RLHF for language models:
Godot RL Agents is an open source package that lets video game creators, AI researchers and hobbyists learn complex behaviors for their Non-Player Characters or agents inside the Godot Engine. It provides an interface between Godot games and ML algorithms running in Python, with wrappers for four well-known RL frameworks — StableBaselines3, CleanRL, Sample Factory and Ray RLlib — plus support for memory-based agents (LSTM / attention), 2D and 3D games, and a suite of AI sensors to augment what the agent can observe. Godot and Godot RL Agents are completely free and open source under a permissive MIT license. Installation is just pip install godot-rl. The library’s creator, Ed Beeching, is a Research Scientist at Hugging Face; see the GitHub page and the AAAI-2022 workshop paper.
Worth remembering from the hands-on (building Ring Pong — pong on a ring, where the objective is to keep the ball bouncing inside the ring — with Godot 4.0):
AIController3D node as a child of the player, then “Extend Script” it (controller.gd) and implement four methods: get_obs(), get_reward(), get_action_space(), set_action(action). GDScript is syntactically similar to Python.[ball_pos.x, ball_pos.z, ball_vel.x/10.0, ball_vel.z/10.0]) and the action space is a single continuous value clamped to $[-1, 1]$.train.tscn) handles Python↔Godot communication over TCP; it exposes a “Speed Up” property (try 8) to accelerate training. Multiple game instances run in parallel for the same reason.gdrl (live in the editor) — a reasonable Ring Pong policy is learned in several minutes.--onnx_export_path=model.onnx, then set the Sync node’s control mode to Onnx Inference and point Onnx Model Path at the file — that runs the agent inside Godot with no Python server. Inference requires the mono (.NET) build of Godot plus .NET.Learning Agents is an Unreal Engine (UE) plugin that allows you to train AI characters using machine learning inside Unreal. It’s the Unreal counterpart of Godot RL Agents / ML-Agents: you build a unique environment with Unreal’s tooling and train agents in it — the course’s worked example is teaching a car to drive in an Unreal environment.
If you’re new to Unreal, the course prescribes a warm-up before touching the plugin:
Then (and this is the whole path if you already know Unreal):
Four environments the course recommends once you want problems harder than the Gym classics:
A software package featuring high-quality environments for RL research, giving a standard interface to popular arcade emulated video games with a Python API fully compliant with the OpenAI Gym/Gymnasium format. Supports Linux, Windows and macOS, installable via pip; free to use, you only need to register on the official website.
A Python library providing a Gym interface to Minecraft, accompanied by datasets of human gameplay (so it’s also an offline-RL / imitation playground). Yearly challenges — see the website.
Donkey is a self-driving-car platform for hobby remote-control cars. This simulator version is built on Unity, using its physics and graphics, and connects to a Donkey Python process so your trained model can control the simulated car.
A famous real-time strategy game; DeepMind used it for Deep RL research with AlphaStar.
(The unit also has a Student Works page collecting projects built by course participants — games and agents in Unity/Pygame/etc., arranged by publication date. You can add your own by opening a PR.)
The question this section addresses: how should we monitor and keep track of powerful RL agents that we are training in the real world and interfacing with humans?
As ML systems have increasingly impacted modern life, the call for documentation of these systems has grown. Such documentation can cover the training data (where it is stored, when it was collected, who was involved) or the model optimization framework (architecture, evaluation metrics, relevant papers) and more. Model cards and datasheets are increasingly available — e.g. on the Hub (model cards documentation); click any popular model on the Hub and you can learn about its creation process.
⚠️ The gap: these model- and data-specific logs are designed to be completed when the model or dataset is created, which leaves them un-updated when those models are built into evolving systems later.
RL systems are fundamentally designed to optimize based on measurements of reward and time. A reward function maps nicely onto well-understood supervised learning (via a loss function), but our understanding of how ML systems evolve over time is limited.
Hence Reward Reports for Reinforcement Learning — the name deliberately mirroring Model Cards for Model Reporting and Datasheets for Datasets. The goal is documentation focused on the human factors of reward and on time-varying feedback systems. Building on the documentation frameworks of model cards (Mitchell et al.) and datasheets (Gebru et al.), the authors argue the need for Reward Reports for AI systems:
Reward Reports are living documents for proposed RL deployments that demarcate design choices.
Many questions remain — applicability across different RL applications, roadblocks to system interpretability, and the resonances between deployed supervised ML systems and the sequential decision-making used in RL. At a minimum, Reward Reports are an opportunity for RL practitioners to deliberate on these questions.
The piece specific to RL and feedback-driven ML is a change-log. It records:
The change-log is accompanied by update triggers that encourage monitoring these effects.
Some of the most impactful RL-driven systems are multi-stakeholder and behind the closed doors of private corporations, which are largely unregulated — so the burden of documentation falls on the public. Reward Reports for popular ML systems are being built as a public record on GitHub; see an example report and the paper.
| Term | One-line definition |
|---|---|
| Online RL | The agent collects its own experience by interacting with the env (directly or via a replay buffer) |
| Offline RL | The policy is learned only from a fixed dataset of other agents’/humans’ trajectories; no interaction |
| Counterfactual queries problem | Offline RL’s core failure: the agent wants to do something the dataset never did (a.k.a. distributional shift / OOD actions) |
| Generalisation gap | Train-level performance minus held-out-level performance |
| Domain randomization | Randomizing env parameters during training to gain robustness / sim-to-real transfer |
| Dynamics model | $s_{t+1} = f_\theta(s_t,a_t)$, learned by minimizing the negative log-likelihood of transitions |
| MPC | Plan over a finite horizon $\tau$ with the model, execute the first action, re-plan |
| Compounding error | Model error accumulating over a multi-step predicted rollout |
| Curriculum Learning | A hand-designed ordering of tasks so skills are acquired incrementally |
| ACL | Mechanisms that automatically adapt the training-task distribution to the agent’s current capabilities |
| Return-to-go | $\hat R_t = \sum_{t’=t}^{T} r_{t’}$ — the conditioning signal of a Decision Transformer |
| Decision Transformer | Autoregressive model over $(\hat R, s, a)$ tokens that generates actions achieving a desired return |
| Tabula rasa | The standard RL setup where the agent learns everything from scratch (no priors) |
| Reward model | $r_\phi(x,y)$, trained on human pairwise comparisons, used as a proxy human inside RL |
| KL penalty ($\beta$) | Term keeping the RL-tuned policy close to the frozen reference model $\pi_{\text{ref}}$ |
| Reward-model over-optimization | Gaining reward-model score while getting worse by real human judgment |
| Reward Report | A living document for an RL deployment, centered on a change-log + update triggers |
AIController3D?Everything below appears in the Bonus Unit 3 pages; grouped for convenience.
Offline RL
Generalisation
Model-based RL
Curriculum learning
Decision Transformers
LMs in RL
RLHF — see the full paper list in §7, starting from Illustrating RLHF.
Game engines
RL documentation
Next up: Bonus Unit 5 — Imitation Learning with Godot RL Agents, where instead of shaping a reward you record your own demonstrations and train a robot to copy them inside a Godot environment — and then the certification page to claim your certificate of completion.