Source: Hugging Face Deep RL Course, Unit 7 — notes from Introduction through Additional Readings, reorganized for review. Continues from Unit 6.
Keypoints:
save_steps, team_change, swap_steps, window, play_against_latest_model_ratio — all trading stability vs diversitypoca trainer + self-play, evaluated on the AI vs AI ELO leaderboardEverything so far has been a single-agent system: our agent was alone in its environment, not cooperating or collaborating with other agents. That works great, and covers many applications (all the CartPole / LunarLander / Pyramids / Doom environments trained so far).
But as humans we live in a multi-agent world — our intelligence comes from interaction with other agents. So the goal becomes: build agents that can interact with other humans and other agents, and are robust enough to adapt, collaborate, or compete.
This unit is the basics of multi-agent reinforcement learning (MARL), and the hands-on is the first genuinely multi-agent training run of the course: a 2v2 soccer team that must beat the opponent team (SoccerTwos, built by the Unity ML-Agents team).
⚠️ Maintenance note from the course itself: the Deep RL course is now in a low-maintenance state, and the AI vs AI leaderboard for SoccerTwos has been shut down. You can still train the agent and watch it play — just don’t expect the ladder to rank you.
In MARL we have multiple agents that share and interact in a common environment — and interact with each other, not just with the world.
Course examples:
The new difficulty is that the other agents are learning too. In single-agent RL the environment’s dynamics $P(s’ \mid s,a)$ are fixed; the moment other learners are in the loop, the thing you are adapting to is itself adapting to you.
Key insight: MARL is not “single-agent RL with more bodies”. The hard part is that the other agents’ policies are part of your transition function, and they change while you learn.
Because agents now interact with each other, the environment can be one of three kinds:
| Cooperative | Competitive / adversarial | Mixed | |
|---|---|---|---|
| Objective | Maximize the common benefit | Maximize own benefit, minimize the opponent’s | Cooperate within a team, compete across teams |
| Reward structure | Shared / team reward | Roughly zero-sum | Team reward, opposed across teams |
| Course example | Warehouse robots loading packages | Tennis (1v1) | SoccerTwos (2v2) |
| Typical tooling | Centralized critic / team reward (MA-POCA) | Self-play + ELO | Self-play + MA-POCA together |
| Evaluation metric | Cumulative team reward | ELO (reward is opponent-dependent) | ELO for the team |
Mnemonic: cooperative = maximize ours; competitive = maximize mine minus theirs; mixed = both, at two different levels.
There are two solutions for designing a MARL system.
In decentralized learning, each agent is trained independently from the others. Each vacuum learns to clean as many places as it can without caring about what other vacuums (agents) are doing.
Why non-stationarity is fatal to the usual guarantees: from agent $i$’s viewpoint the transition and reward it experiences are really marginals over everyone else’s policies $\pi_{-i}$,
\[P_i\!\left(s' \mid s, a_i\right) \;=\; \sum_{a_{-i}} P\!\left(s' \mid s, a_i, a_{-i}\right) \prod_{j \neq i} \pi_j\!\left(a_j \mid s\right)\]and since $\pi_{-i}$ is changing during training, $P_i$ is a moving target. The convergence proofs behind Q-Learning and friends assume a fixed MDP, so they simply don’t apply: the agent can end up chasing its own tail, and old replay-buffer transitions become stale/off-distribution rather than merely off-policy.
Here a high-level process collects all the agents’ experiences — a shared experience buffer — and those experiences are used to learn a common policy.
In the vacuum cleaner example, the (joint) observation would be:
We use that collective experience to train a policy that moves all three robots in the most beneficial way as a whole, so each robot learns from the common experience. Crucially, we now have a stationary environment, because all the agents are treated as one larger entity and the changes in other agents’ policies are known (they are the same policy).
| Decentralized | Centralized | |
|---|---|---|
| Who learns | Each agent independently | One policy learned from all agents |
| What other agents are | Just part of the environment | Part of a single larger entity |
| Input / output | Local observation → own action | Present state of the environment → joint actions |
| Reward | Individual / local | Global |
| Stationarity | Non-stationary ⇒ no convergence guarantee | Stationary |
| Scaling | Scales easily, no communication needed | Joint action space grows exponentially in the number of agents; needs shared info at execution time |
| Reuse of single-agent algos | Direct | Needs multi-agent machinery |
Key insight: the decentralized/centralized choice is a trade between convenience and correctness. Decentralized is trivially easy to implement but breaks the stationarity assumption; centralized restores stationarity but needs global information (which a real robot fleet may not have at run time).
The practical compromise, and the one the hands-on actually uses: centralized training, decentralized execution (CTDE). Train with privileged global information, then throw it away at deployment.
That’s exactly what MA-POCA (Multi-Agent POsthumous Credit Assignment), the ML-Agents multi-agent trainer, does. The problem it solves: agents typically receive a reward as a group (+1 minus penalty) when the team scores, so every agent is rewarded even if it didn’t contribute equally to the win — which makes it hard to learn what to do individually (the credit-assignment problem).
The idea is simple but powerful: a centralized critic processes the states of all agents in the team to estimate how well each agent is doing — think of this critic as a coach. This lets each agent
The “posthumous” part is the neat trick: the critic can still assign credit to an agent whose action mattered even if that agent is removed from the episode before the reward arrives (absorbing states) — see the paper in §9.
Training agents correctly in an adversarial game is quite complex, and it’s a chicken-and-egg problem:
The course’s analogy: a child who just started learning soccer. Playing against a very good player is useless — it’s too hard to win, or even to get the ball from time to time. The child continuously loses without ever having time to learn a good policy. The symmetric failure also holds:
So a fixed random opponent is not a shortcut: you will get an agent that is excellent at beating noise and hopeless against anyone with a strategy.
The best solution is to have an opponent that is on the same level as the agent, and that upgrades its level as the agent upgrades its own.
That solution is self-play: the agent uses former copies of itself (of its policy) as an opponent.
This way the agent
It is a way to bootstrap an opponent and progressively increase the opponent’s complexity — an automatic curriculum that you never have to hand-design. It mirrors how humans learn in competition: start against a similar-level opponent, learn from them, then move on to stronger opponents.
The loop, in one line:
\[\pi_{\theta} \;\text{trains against}\; \{\pi_{\theta_{t_1}}, \pi_{\theta_{t_2}}, \dots\} \;\subset\; \text{snapshot pool} \quad\Longrightarrow\quad \text{snapshot } \pi_{\theta} \text{ every } \texttt{save\_steps} \;\Longrightarrow\; \text{pool improves with } \pi_{\theta}\]The theory is not new. It was already used by Arthur Samuel’s checkers player in the fifties and by Gerald Tesauro’s TD-Gammon in 1995.
Self-play is built into ML-Agents and controlled by several hyperparameters. The documentation states the central tension explicitly:
The tradeoff is between the skill level and generality of the final policy, and the stability of learning.
Training against a set of slowly changing or unchanging adversaries with low diversity results in more stable training — but there is a risk of overfitting if the change is too slow. Conversely, quickly changing, highly diverse adversaries give a more general policy but a noisier, less stable learning process.
So there are four things to control: how often we change opponents, how many opponents we keep, how often we face the newest one, and how often we snapshot.
| Hyperparameter | Unit | What it does | Larger value ⇒ | Trade-off it controls | SoccerTwos value |
|---|---|---|---|---|---|
save_steps | trainer steps | Number of training steps before saving a new opponent (a snapshot of the current policy) | Snapshots are further apart in skill, so the pool covers a wider range of skill levels and play styles (each snapshot got more training). Harder problem ⇒ may need more total steps, but a more general, robust final policy | Snapshot granularity: fine-grained-and-similar vs coarse-and-varied opponents | 50000 |
team_change | trainer steps | Number of steps between switching which team is the learning team | The agent trains longer against the same set of opponents, so it gets better at defeating them — but too long risks overfitting to those particular strategies, and it may fail against the next batch | Depth vs breadth of adaptation; also determines how many snapshots get saved per switch (docs recommend setting it as a multiple of save_steps, typically 4×–10×) | 200000 |
swap_steps | ghost steps | Number of steps between swapping the opponent’s policy for a different snapshot from the pool. (A ghost step is a step by an agent following a fixed policy and not learning — the distinction matters in asymmetric games such as 2v1, where the two teams collect agent-steps at different rates) | The agent plays the same fixed opponent for longer ⇒ more stable training, but risks overfitting its behavior to that particular opponent, so it may lose more than expected right after a swap | Within-session opponent churn rate (stability vs not overfitting) | 2000 |
window | snapshots | Size of the sliding window of past snapshots from which opponents are sampled; when a new snapshot is taken, the oldest is discarded | The pool contains a larger diversity of behaviors, since it keeps policies from earlier in the training run — more general policy, harder problem | Opponent-pool diversity (and protection against cycling/forgetting how to beat old strategies) | 10 |
play_against_latest_model_ratio | probability | Probability of playing against the latest opponent policy; with probability $1-\rho$ the agent plays a snapshot from a past iteration | Plays the current opponent more often. Since that opponent keeps updating, this is an unstable learning environment — but it poses an auto-curriculum of increasingly challenging situations, which may give a stronger final policy | Freshness vs stability: chase the moving target, or grind against a frozen history | 0.5 |
initial_elo | rating | Starting ELO for the self-play bookkeeping | — | — | 1200.0 |
Typical ranges from the docs: save_steps 10k–100k, swap_steps 10k–100k, window 5–30, play_against_latest_model_ratio 0.0–1.0, team_change 4×–10× save_steps.
swap_steps formula for asymmetric teams. If you want $x$ swaps for a team of num_agents against a team of num_opponent_agents over team_change total steps:
Worked example from the docs (2v1, $x=4$ swaps over team_change $=200000$): the team of one agent gets $\tfrac{1}{2}\times\tfrac{200000}{4} = 25000$; the team of two gets $\tfrac{2}{1}\times\tfrac{200000}{4} = 100000$. With equal team sizes the first factor is 1, so swap_steps is just total steps ÷ desired number of swaps.
Key insight: every one of these knobs is the same dial in disguise — stability of learning vs diversity/generality of opponents. Slow, static, latest-only opponents ⇒ stable but overfit. Fast, wide-window, snapshot-sampled opponents ⇒ general but noisy.
Note on reward signals for self-play. ML-Agents assumes the final reward in a trajectory encodes the episode outcome: +1 = win, −1 = loss, 0 = draw (the ELO calculation depends on this). The docs advise being conservative when shaping rewards in adversarial games, because of the instability and non-stationarity — start with the simplest possible reward function (+1 win, −1 loss) and allow more training iterations to compensate for its sparsity.
In adversarial games, tracking cumulative reward is not always a meaningful metric of learning progress, because it depends only on the skill of the opponent. An agent scoring +0.9 per episode against a weak opponent has told you nothing about how good it is; and in self-play the opponent is changing under you, so the reward curve is measuring two moving things at once.
Instead we use the ELO rating system (named after Arpad Elo), which computes the relative skill level between 2 players from a given population in a zero-sum game.
| Cumulative reward | ELO rating | |
|---|---|---|
| Measures | Absolute performance in a fixed environment | Relative skill against a population |
| Depends on the opponent? | Yes — entirely (that’s the problem) | Yes, and it corrects for it (opponent’s rating is in the formula) |
| Interpretable across training? | Not in self-play — the opponent improves too | Yes — should rise as the agent gets stronger |
| Good for | Cooperative / single-agent tasks | Adversarial tasks, ladders, leaderboards |
| Team games | Team reward is fine | Works — average the team’s ratings and use that |
| Failure mode | Meaningless if opponent strength varies | Deflation; ignores individual contribution; not comparable across eras |
Zero-sum means one agent wins and the other loses: each participant’s gain or loss of utility is exactly balanced by the gain or loss of the other participants, so the sum of utility equals zero.
initial_elo: 1200.0 in the config). It can decrease initially but should increase progressively during training.After every game, the winner takes points from the loser, and the number of points is set by the difference in the two ratings (hence relative):
| Outcome | Points transferred |
|---|---|
| Higher-rated player wins | Few points taken from the lower-rated player |
| Lower-rated player wins | A lot of points taken from the higher-rated player |
| Draw | The lower-rated player gains a few points from the higher |
For players $A$ and $B$ with ratings $R_A$ and $R_B$, the expected scores are
\[E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}, \qquad E_B = \frac{1}{1 + 10^{(R_A - R_B)/400}}\]so that $E_A + E_B = 1$ (a 400-point gap corresponds to 10:1 expected odds). Then, at the end of the game, we update the actual rating with a linear adjustment proportional to the amount by which the player over- or under-performed:
\[\boxed{\,R_A' = R_A + K\,\bigl(S_A - E_A\bigr)\,}\]Because $E_A + E_B = 1$ and $S_A + S_B = 1$, the two updates are exactly opposite: $K(S_A - E_A) = -K(S_B - E_B)$. The total rating in the pool is conserved — that’s the zero-sum property showing up in the bookkeeping.
Player A rated 2600, player B rated 2300, and $K = 16$.
Expected scores:
\[E_A = \frac{1}{1 + 10^{(2300 - 2600)/400}} = 0.849, \qquad E_B = \frac{1}{1 + 10^{(2600 - 2300)/400}} = 0.151\]If A wins ($S_A = 1$, $S_B = 0$):
\[\text{ELO}_A = 2600 + 16 \times (1 - 0.849) = 2602, \qquad \text{ELO}_B = 2300 + 16 \times (0 - 0.151) = 2298\]If B wins ($S_A = 0$, $S_B = 1$):
\[\text{ELO}_A = 2600 + 16 \times (0 - 0.849) = 2586, \qquad \text{ELO}_B = 2300 + 16 \times (1 - 0.151) = 2314\]Note the asymmetry, which is the whole point: the favourite winning moves the ratings by ≈2 points, the underdog winning moves them by ≈14.
Advantages
Disadvantages
poca, and AI vs AISoccerTwos (Unity ML-Agents team): four agents compete in a 2v2 toy soccer game. The goal is to get the ball into the opponent’s goal while preventing the ball from entering your own goal — i.e. the mixed cooperative/competitive setting. There are two Multi-Agent Groups with two agents each, and the provided executable contains 8 copies of SoccerTwos running in parallel.
| Piece | Spec |
|---|---|
| Observation space | Vector of size 336: 11 ray-casts forward over 120° (264 dims) + 3 ray-casts backward over 90° (72 dims), each detecting 6 object types — Ball, Blue Goal, Purple Goal, Wall, Blue Agent, Purple Agent — plus the object’s distance |
| Action space | 3 discrete branches — forward/backward movement, sideways movement, and rotation |
| Visual observations | None |
| Reward (team-dependent) | $(1 - \text{accumulated time penalty})$ when the ball enters the opponent’s goal, where the penalty is incremented by $1/\texttt{MaxStep}$ every fixed update and reset to 0 at episode start; $-1$ when the ball enters your own goal |
So the reward is a team reward — it lands on both teammates regardless of who actually did the work. That’s precisely the credit-assignment problem MA-POCA exists for (§3.4). The recipe is therefore: self-play (to beat the other team) + the MA-POCA trainer (to learn cooperation inside your team). In the config that trainer is called poca.
./config/poca/SoccerTwos.yaml — the parts worth remembering:
behaviors:
SoccerTwos:
trainer_type: poca
hyperparameters:
batch_size: 2048
buffer_size: 20480
learning_rate: 0.0003
beta: 0.005 # entropy regularization
epsilon: 0.2 # PPO-style clipping
lambd: 0.95 # GAE lambda
num_epoch: 3
learning_rate_schedule: constant
network_settings:
normalize: false
hidden_units: 512
num_layers: 2
vis_encode_type: simple
reward_signals:
extrinsic:
gamma: 0.99
strength: 1.0
keep_checkpoints: 5
max_steps: 5000000
time_horizon: 1000
summary_freq: 10000
self_play:
save_steps: 50000
team_change: 200000
swap_steps: 2000
window: 10
play_against_latest_model_ratio: 0.5
initial_elo: 1200.0
Training is launched with mlagents-learn <config> --env=<executable> --run-id="SoccerTwos" --no-graphics, and pushed with mlagents-push-to-hf --run-id=... --local-dir=./results/<run_id> --repo-id=<user>/<repo> --commit-message=....
Practical notes from the course:
Ctrl-C only once to stop training: ML-Agents needs to write the final .onnx file before closing.AI vs AI is the open-source tool built for this unit to compete agents on the Hub against one another in a multi-agent setting and rank them on a leaderboard. The point is robust evaluation: by evaluating your agent against many others, you get a good idea of the quality of your policy — exactly the argument for ELO over reward.
It is three tools:
The loop: every four hours the algorithm fetches all available models for the environment (selected by the ML-Agents-SoccerTwos tag, with a SoccerTwos.onnx file in the repo), builds a match queue, simulates each match in a headless Unity process recording 1 / 0.5 / 0 for win / draw / loss, and when the queue is done updates every model’s ELO and the leaderboard. Note that those three result values are exactly the $S_A$ of §6.3 — the ladder is literally running the ELO update above.
⚠️ Matches you watch live in the demo Space are not counted toward your rating; they’re only for visualization. And per the maintenance notice, the leaderboard itself is now offline.
save_steps (snapshot spacing), team_change (which team learns), swap_steps (opponent churn), window (pool size), play_against_latest_model_ratio (latest vs historical).window, play_against_latest_model_ratio, save_steps, swap_steps + team_change.initial_elo of 1200 in the first ~2M steps?These are optional if you want to go deeper.
An introduction to multi-agents
Self-Play and MA-POCA
Reference docs used in the hands-on
Next up: Proximal Policy Optimization (PPO) — deriving the clipped surrogate objective that keeps policy updates inside a trust region, the workhorse behind (among many other things) the
epsilon: 0.2we just copied into the SoccerTwos config.