Centralized Training and Decentralized Execution in Multi-Agent RL

The agent that acts at deployment is not the agent that learns. Training sees a richer world than execution ever will.

That asymmetry feels like a violation of good engineering instincts. In supervised learning, you train on the inputs the model will receive at inference time—and that rule serves you well. Multi-agent reinforcement learning breaks it on purpose. Understanding why is the difference between treating centralized training with decentralized execution as a clever trick and treating it as what it actually is: a deliberate information contract between two regimes.

The Information Asymmetry at the Heart of CTDE

Centralized training with decentralized execution (CTDE) splits the learning problem into two distinct information regimes. During training, a centralized component may see the global state, the joint observations of every agent, or the actions other agents took. At execution, each agent selects actions using only its own local observation history. No global state. No teammates' observations. No centralized critic.

This is not an implementation shortcut. It is a design choice: you spend information during training to buy a better policy, then discard that information before deployment.

To see why anyone would do this, consider the two poles CTDE sits between. Fully decentralized training lets each agent learn alone with its own reward signal. It is simple and deployable, but it suffers from non-stationarity: from any single agent's perspective, the environment changes as teammates update their policies. Fully centralized control—one policy that outputs joint actions for everyone—avoids that instability but requires global observation and communication at every decision step. That is impractical at scale, and it creates a central point of failure.

CTDE takes the middle path. Extra training-time information stabilizes learning and eases credit assignment. Execution stays cheap and local, with no dependence on a central coordination service at decision time.

Note: The mental model to hold onto: the critic shapes the policy's gradient, not its inputs. The policy learns from a teacher that saw everything, then performs alone with almost nothing.

Why Decentralized Training Fails: Non-Stationarity and Credit Assignment

If you have worked through multi-agent RL fundamentals, you already know the core problem: another learning agent makes the environment non-stationary from your perspective. The transition dynamics shift as teammates update their policies, so an independent learner chases a moving target. Its value estimates oscillate. Convergence becomes a hope rather than a guarantee.

The second failure is credit assignment. When a team reward arrives, an independent learner cannot tell which of its own actions produced it and which were caused by teammates. It updates as if it were solely responsible, which corrupts the value estimate with noise it cannot resolve.

A centralized critic or joint value function sidesteps both problems by conditioning on the global state and joint actions. Teammates are no longer unmodeled noise in the environment—they are explicit inputs to the value estimate. The critic can answer the question independent learning cannot: given everything that actually happened, how good was this joint action?

That is the entire motivation for CTDE. The centralized component exists to remove the two failure modes that make independent learning unreliable, and it can only do that by seeing what no single agent sees.

Tracing One Training Step: Where the Information Flows

The mechanism matters more than the label, so let us trace one cooperative timestep end to end.

Each agent observes its own局部 view of the world and samples an action from its decentralized policy. The agents act simultaneously. The environment advances and returns a team reward.

Now the centralized component enters. It receives the global state, the joint actions, and the reward. For a centralized critic, that typically means concatenating per-agent states into a fixed-dimension input and feeding it to the critic network. If the agents are homogeneous, you can permute the input order to increase robustness—the critic should not care which agent occupies which slot. If you want the critic to weight some agents' states more heavily than others, attention mechanisms can learn that weighting explicitly.

The critic produces a value estimate conditioned on information no single agent will have at execution. That estimate—or the advantage derived from it—shapes the policy update for each agent. The critic evaluates the joint state rather than guessing at it from a partial view.

Here is the boundary you must keep visible: the critic's global knowledge enters the update, not the action. The next time the agent selects an action, its policy consumes only its own local observation. Centralized training cannot manufacture information that is absent from the actor's deployment observation. If the local observation does not contain enough signal to support the coordinated behavior, the policy simply cannot learn to execute it reliably—no matter how good the critic was.

The practical cost of this flow is real. Gathering and processing global information during training is a computational and bandwidth burden that grows with agent count. The centralized critic's input dimension scales with the number of agents, and selecting joint actions centrally would face an action space that grows exponentially with agent count. CTDE does not eliminate the curse of dimensionality—it moves it into the training phase, where you have the resources to absorb it.

Two Ways to Spend the Extra Information

The CTDE framework contains two dominant architecture families. They differ in what the centralized component learns and how it guides the decentralized policies.

Centralized Critic (Actor-Critic Family)

Each agent keeps its own policy, but a shared or per-agent critic conditions on the global state and joint actions to produce a lower-variance advantage estimate. If you understand actor-critic methods, the extension is natural: the critic guides the policy without becoming the policy. At execution, the critic is absent from the action path entirely—the actor's inputs never included it.

The centralized critic generalizes naturally to mixed or competitive settings because it does not assume a shared team reward. Each agent can have its own critic that happens to see everyone's information.

Value Decomposition (Q-Learning Family)

A joint Q-function over the global state is trained centrally, then factored into per-agent utilities that each agent can maximize from its own observation. VDN and QMIX are the canonical examples: they learn a joint action-value and decompose it into individual components under constraints that preserve the argmax.

Value decomposition suits cooperative tasks with a shared team reward, where the factorization can be designed to guarantee that maximizing individual utilities also maximizes the joint value.

The shared pattern matters more than the family names: in both architectures, the centralized component is a training-time scaffold. But the execution artifact differs. Centralized-critic methods leave behind trained actor policies. Value-decomposition methods leave behind per-agent utility functions that support decentralized greedy action selection. "Discard the centralized component" is a deployment summary, not an identical implementation step.

What Decentralized Execution Actually Requires

The execution contract is plain: each agent selects actions from its own policy using only its local observation history. No global state. No joint actions. No centralized critic. The centralized component must be fully absent from the action path.

This is a stronger requirement than it sounds. You cannot keep the centralized critic in a degraded form—say, running it on a server and broadcasting value estimates to agents. That would reintroduce the communication costs and single-point-of-failure risks CTDE was designed to avoid. The critic is not a deployment artifact. It is a training instrument.

Common mistake: Treating the centralized critic as something you can query at execution time when an agent is uncertain. If execution depends on the critic, you no longer have decentralized execution. You have a fragile distributed system.

CTDE fits when training-time global information is available but deployment-time communication is costly or unreliable. It is overkill when agents already share full state, or when a single centralized controller is genuinely feasible.

One clarification keeps the contract honest: CTDE removes dependence on a central execution service, but it does not remove all communication or coordination failure. If your execution contract allows local message exchange between agents, that communication is part of the decentralized policy's inputs—and its loss is a real fault condition. If your contract is strictly observation-only, then communication was never part of the system. Know which contract you designed before you test it.

Leakage and Train-Deploy Mismatch: Where CTDE Breaks

CTDE's information asymmetry is its strength and its vulnerability. The failure modes are silent: the policy looks excellent during training evaluation and collapses at deployment.

Leakage is any information that reaches the agent's action-selection inputs at execution but was not part of the intended local observation. A concrete example: an agent whose policy accidentally conditions on a teammate's hidden state through a shared feature. If that feature is available during training but absent at deployment, the policy has learned to rely on something it cannot sense.

The subtler mismatch is harder to catch. Even without explicit leakage, a policy trained under a centralized critic can overfit to coordination patterns that only emerge because the critic saw everything. The policy learns to act as if its teammates will behave in ways that were only possible because the critic's global view shaped the training dynamics. At execution, those patterns are absent, and the policy fails without understanding why.

There is also a structural critique worth knowing: standard CTDE makes an independence assumption on agent policies. Each policy conditions only on its own observation, which limits how much global cooperative information actually shapes training. Recent research argues this means CTDE cannot fully utilize global information, leading to inefficient joint exploration. The framework buys stability, but it does not buy true joint policy optimization.

Warning: Before deploying a CTDE-trained policy, audit what each agent's action inputs actually contain. Then test the policy under degraded conditions to expose hidden dependencies. If the policy collapses when a channel disappears, you trained a dependent system, not a decentralized one.

A Practical Audit Before You Trust a CTDE Policy

The conceptual material only earns its keep if it changes how you validate a trained system. Run this checklist before you trust any CTDE policy in deployment.

Audit the observation space. Confirm each agent's execution inputs contain only what it can actually sense. Inspect the policy network's input features directly. If any feature encodes global state, teammate hidden state, or information that will not exist at deployment, you have leakage.

Inspect the deployment graph. Trace the actual execution path and prove that no critic output is consumed by the actor. This is an architectural check: the critic should not appear in the deployment graph at all. If it does, you have not built decentralized execution.

Test under degraded conditions. Run the trained policies with communication dropped or observations masked. Graceful degradation is the goal. If performance collapses when a channel disappears, the policy was depending on it. Note that this test checks behavioral dependence, not architectural presence—a policy can fail without the critic ever appearing in its execution graph.

Compare against a decentralized baseline. Train an independent learner on the same task. If the CTDE policy does not beat it, the extra training complexity bought nothing. This comparison is the honest test of whether the centralized information actually reduced coordination difficulty.

The decision rule: CTDE earns its complexity only when the training-time information genuinely reduces coordination difficulty, and the execution-time policy survives without it. Both conditions must hold. The first condition justifies the training cost. The second condition determines whether the trained system will work in the real world.

The Information Contract

CTDE is not a training trick. It is a deliberate information contract between two regimes: training may see everything, execution may see almost nothing, and the bridge between them is the gradient signal that shapes decentralized policies from centralized knowledge.

The contract works when you respect both sides. Spend global information during training only when it buys coordination. Audit the execution-time policy for hidden dependencies before trusting it. And when the audit fails, do not patch the deployment—fix the training asymmetry that created the dependency.

The natural next step is to build a small centralized-critic actor-critic setup on a cooperative benchmark and run this audit checklist against it. Watch the policy learn with the critic's help, then watch what happens when you test under degraded conditions. The failure modes you find will teach you more about information boundaries in multi-agent RL than any architecture diagram can.