advanced
9 min read
Multi-Agent Reinforcement Learning Explained: What Changes When Other Agents Learn Too?
Single-agent RL treats the environment as a fixed target. Add a second learning agent, and the target starts moving.
Read tutorialadvanced
Multi-Agent Reinforcement Learning
Discover what changes when agents share the world: cooperate, compete, communicate, model opponents, assign credit, and improve through self-play.
Tutorials
Articles in Multi-Agent Reinforcement Learning
Follow the learning path in order or jump directly to the RL concept or experiment you need.
advanced
11 min read
Centralized Training and Decentralized Execution in Multi-Agent RL
The agent that acts at deployment is not the agent that learns. Training sees a richer world than execution ever will.
Read tutorial
advanced
11 min read
Non-Stationarity in Multi-Agent Reinforcement Learning: Why Yesterday’s Target Moves
Your training curves look beautiful. Reward is climbing, losses are shrinking, and the agent seems to have mastered its task. Then you deploy it against a…
Read tutorial
advanced
11 min read
Self-Play in Reinforcement Learning: How Agents Learn From Evolving Opponents
The opponent is part of the environment, and the opponent keeps learning. That single fact turns self-play into a moving-target problem where your own…
Read tutorial
advanced
11 min read
Credit Assignment in Multi-Agent RL: Which Agent Caused the Outcome?
Your team just won. The shared reward arrives, positive and unambiguous. Every agent updates its policy as if the victory were its own doing—and that is…
Read tutorial
advanced
11 min read
Value Decomposition in Multi-Agent RL: Learning Team Actions From Local Views
A team shares one reward, but at execution time each agent must pick its own action from only its own observation. Value decomposition answers that…
Read tutorial
advanced
13 min read
Opponent Modeling in Multi-Agent RL: Predicting the Other Learner
The moment you treat another learning agent as part of the environment, you are betting on a target that moves. Opponent modeling is the attempt to stop…
Read tutorial
advanced
8 min read
Communication and Coordination in Multi-Agent RL
More communication is not automatically better coordination. In cooperative multi-agent reinforcement learning, a naive message-passing channel can inject…
Read tutorial
advanced
11 min read
Cooperative vs Competitive Multi-Agent RL: How Incentives Change Learning
"Cooperative" and "competitive" describe the reward wiring, not the environment—and the same physical scene can host either regime depending on how you…
Read tutorial
advanced
11 min read
Evaluating Multi-Agent Policies: Partners, Opponents, and Exploitability
A policy that crushes its self-play twin can still collapse against a slightly different opponent. That single blind spot is why multi-agent reinforcement…
Read tutorial
advanced
10 min read
Why Monotonic Value Factorization Supports Decentralized Greedy Actions
A team where every agent picks its own best action can still lose. Monotonic value factorization is the constraint that rules out one specific failure —…
Read tutorial
advanced
11 min read
Counterfactual Credit Assignment in Multi-Agent RL: A Worked Derivation
A team wins. The shared reward is +1. Every agent's gradient gets the same number, so the agent who stood still learns exactly as much as the agent who…
Read tutorial
advanced
13 min read
Markov Games and Moving Learning Targets in Multi-Agent RL
The game never changes. Your experience of it changes every time another agent updates.
Read tutorial
advanced
9 min read
Test Self-Play for Cycling and Exploitability in a Small Game
A self-play win rate can climb for a thousand iterations and still be lying to you. Here is the smallest instrument that catches it.
Read tutorial
advanced
10 min read
Audit Centralized Training and Decentralized Execution in a Toy Task
A CTDE run can post a beautiful training curve while the deployed policy quietly cheats. The critic saw the joint state. So did the actor. Nobody noticed,…
Read tutorial
advanced
11 min read
Test a Simple Opponent Model in a Repeated Game
An adaptive agent that beats a fixed opponent in one run can lose to the same opponent after a single behavior shift. The frequency estimate it trusts is a…
Read tutorial
advanced
9 min read
Compare Shared and Counterfactual Credit Signals in a Toy Team Task
Two agents, one shared reward, and a number that refuses to say who earned it. The fastest way to understand that refusal is to force it into arithmetic:…
Read tutorial
advanced
9 min read
Analyze Cooperation, Competition, and Mixed Motives With Payoff Matrices
Two agents converge. Both policies are stable. Both agents are, by their own local test, optimal. The team still loses.
Read tutorial
advanced
8 min read
What One Bit of Communication Enables in a Two-Agent Coordination Game
Two agents want the same thing. Both know the rules. Both are fully cooperative. And they still fail half the time — until someone is allowed to say a…
Read tutorial
advanced
9 min read
Test How Message Bandwidth Changes Multi-Agent Coordination
A team that coordinates perfectly with unlimited chatter can fall apart the moment you cap messages per step or flip a few bits. That collapse is the…
Read tutorial
advanced
11 min read
Test Value Factorization in a Small Multi-Agent Coordination Game
Two agents. Three actions each. Nine joint outcomes. If you cannot find the coordination failure in a table that small, you will not find it in a training…
Read tutorial
advanced
9 min read
Measure Stale Experience When an Opponent Policy Changes
A return drop after an opponent switches is not evidence of a moving target. It is evidence that you have not yet measured the noise floor.
Read tutorial
advanced
9 min read
Formalize Information Constraints in Centralized Training and Decentralized Execution
Your team scores well in the training harness. You strip the centralized critic, deploy the actors, and the score collapses — even though the environment…
Read tutorial
advanced
12 min read
Update a Bayesian Model of an Opponent’s Actions
Frequency counting answers with a single number and no error bar. It cannot tell you how much evidence you actually have, and it has no place to put what…
Read tutorial
advanced
10 min read
Exploitability in Multi-Agent Reinforcement Learning: Best Responses and a Worked Game
A policy that wins 90% of its self-play games can still lose to an opponent that never appeared in training.
Read tutorial
advanced
13 min read
Test Prior Sensitivity in a Bayesian Opponent Model
Same opponent. Same six observed actions. Same code. Two different decisions — because the prior changed.
Read tutorial