Audit Centralized Training and Decentralized Execution in a Toy Task

A CTDE run can post a beautiful training curve while the deployed policy quietly cheats. The critic saw the joint state. So did the actor. Nobody noticed, because the numbers looked fine.

This is a practice tutorial, so the deliverable is not a checklist. It is a small cooperative task you can actually run, instrument, and break on purpose, then read the resulting action and performance differences. The audit question is narrow: what tensors does each agent's action-selection path touch at execution time?

That distinction matters because three different failure classes look identical on a reward plot:

• Actor consumes global state. The actor's input vector includes the joint state or a concatenation of all agents' observations.
• Actor consumes other agents' observations or actions. Even without the full state, the actor reads a teammate's local view or last action, which will not exist at execution.
• Actor consumes a critic-derived signal. The actor's input is local, but it includes a feature produced by a critic that itself depends on unavailable information.

Only the first two are obvious in code review. The third survives most audits, because the actor's declared input shape looks clean.

Legitimate centralized information lives in exactly two places: the training-time critic, or a mixing network that combines per-agent values. Both must be discarded before execution. If a centralized tensor reaches the action-selection path, you have leakage, regardless of how the run performs. If you want the underlying motivation for why centralized training helps at all, the prerequisite article on CTDE information flow covers it. Here we assume you accept the paradigm and want to verify your implementation respects it.

The Toy Task and Its Information Contract

Dependencies: Python 3 and numpy. No GPU, no framework, deterministic seeding. I keep the learner hand-rolled because a framework's abstractions hide exactly the tensors you are trying to inspect.

The task: three agents on a 5×5 grid, each with a local partial observation, a shared team reward, and one global state vector available only during training. Small enough that a full run finishes in seconds, because the audit loop is the point, not the training.

import numpy as np

N_AGENTS = 3
GRID = 5
N_ACTIONS = 5  # up, down, left, right, stay

def reset(rng):
    positions = rng.integers(0, GRID, size=(N_AGENTS, 2))
    target = rng.integers(0, GRID, size=2)
    return positions, target

def local_obs(positions, target, i):
    # agent i sees only its own position and the target
    return np.array([*positions[i], *target], dtype=np.float32)

def global_state(positions, target):
    # available ONLY during training
    return np.array([*positions.flatten(), *target], dtype=np.float32)

def step(positions, target, actions):
    moves = np.array([[0,0],[0,-1],[0,1],[-1,0],[1,0]])
    positions = np.clip(positions + moves[actions], 0, GRID - 1)
    reward = -np.mean(np.linalg.norm(positions - target, axis=1))
    return positions, reward

The information contract, written as a spec you can assert against:

• Each agent may read local_obs(positions, target, i) and its own index i.
• The trainer may read global_state(positions, target) and all agents' actions.
• The execution path may not read global_state, any local_obs(j) for j != i, or any tensor derived from either.

One transition tuple looks like (obs_i, action_i, reward, next_obs_i, global_state, next_global_state). The last two fields exist for the critic only. If your environment returns them in the same dict the actor consumes, you have already blurred the boundary.

Instrument the Execution Path Before You Train

Build the audit harness first. Leakage detected on update one is a five-minute fix; leakage detected after a suspicious result is an afternoon of archaeology.

The core idea: wrap action selection so it records which arrays it received, not which arrays exist in scope. The guard below tracks provenance by object identity, which is enough for direct inputs and for arrays that were explicitly tagged when created. It does not automatically follow a value through arithmetic, so any derived feature you feed the actor must be tagged at the point of creation.

CENTRALIZED = set()

def mark_centralized(arr):
    CENTRALIZED.add(id(arr))
    return arr

def act(policy, obs_i, agent_index, audit_log, execution=True):
    inputs = {"obs": obs_i, "agent_index": agent_index}
    if execution:
        for name, arr in inputs.items():
            if id(arr) in CENTRALIZED:
                raise RuntimeError(
                    f"centralized tensor reached actor via '{name}'"
                )
    audit_log.append({k: v.shape for k, v in inputs.items()})
    return policy(obs_i, agent_index)

Two design choices carry the weight here.

First, separate the training forward pass from the execution forward pass as two distinct functions. If they share a code path, the guard has nothing to check, because the same function legitimately receives global state during training.

Second, mark provenance at creation, not at use. Call mark_centralized the moment global_state is produced, so any array that carries it forward and is itself tagged will be caught. This is the honest boundary of the mechanism: identity tagging catches direct inputs and explicitly tagged derivatives, and it will miss a centralized value that was silently transformed into an untagged array.

Common mistake: Reasoning about leakage by reading the function signature. A variable being visible inside the function is not the same as the policy depending on it. The guard checks what was actually passed, which is the only thing that determines behavior.

Run the Clean Baseline and Read the Output

Train the correctly wired agent. Record three things: team return, per-agent action distributions, and the audit log.

Expected observable: the guard never fires. The audit log shows execution inputs containing only obs and agent_index, with obs shaped (4,) for this task.

# after a clean run
assert all(set(entry.keys()) == {"obs", "agent_index"} for entry in audit_log)
assert all(entry["obs"] == (4,) for entry in audit_log)

Interpret the learning curve as a sanity check, not as proof of correctness. A correct run and a leaky run can both improve. The curve tells you the wiring is not catastrophically broken; it does not tell you the wiring is honest.

Signals worth keeping for comparison: action entropy over time, coordination rate (fraction of steps where agents move toward the target), and whether agents specialize into distinct roles. These become your reference fingerprints when you inject a leak.

Note: A state-based critic can be unsound in partially observable domains. A clean audit does not guarantee a good policy, only an honest one. A state-based critic cannot represent the value of information-gathering actions, so it will undervalue listening. Your audit checks the information boundary, not the critic's representational adequacy.

Inject the Leak and Watch What Changes

Three variants, each isolating a different failure class.

Variant A: Global State in the Actor

Concatenate global_state into the actor's input in both phases.

def act_leaky_a(policy, obs_i, global_state, agent_index, audit_log):
    inputs = {"obs": obs_i, "global_state": global_state}
    for name, arr in inputs.items():
        if id(arr) in CENTRALIZED:
            raise RuntimeError(f"centralized tensor via '{name}'")
    return policy(np.concatenate([obs_i, global_state]), agent_index)

The guard fires immediately. This is the easy case: the leak is loud.

Variant B: Critic-Derived Feature in the Actor

Keep the actor's declared input local, but feed it a feature the critic computed from joint information.

def act_leaky_b(policy, obs_i, critic_feature, agent_index, audit_log):
    # critic_feature was computed from global state; tag it at creation
    return policy(np.concatenate([obs_i, critic_feature]), agent_index)

If critic_feature was tagged when the critic produced it, the guard fires. If it was not tagged, the guard stays silent and you have found the gap in your instrumentation. This variant is the reason provenance marking must happen at creation.

Variant C: Deployment Mismatch

Train with global state in the actor, then strip it at execution.

# training: actor sees [obs_i, global_state]
# execution: actor sees [obs_i] only

The guard does not fire, because execution inputs are genuinely local. But the actor learned a conditional policy on information that will not exist later. Watch the team return: it climbs during training, then falls off a cliff the moment you remove the global state. The action distribution shifts abruptly at the boundary.

The mechanism is off-distribution execution. The actor's weights encode "if the target is at (3,4) and my teammate is at (1,1), move right." At execution it only sees its own position and the target, so the conditional it learned is evaluated on inputs it never saw during training. The policy is not wrong; it is answering a different question.

Diagnose the Signature, Not Just the Score

A single metric cannot distinguish leakage from ordinary instability. The signatures differ in shape.

The decision rule: if the audit log shows a centralized tensor in the execution path, fix the wiring before tuning anything else. Hyperparameter search on a leaky setup optimizes the leak, not the policy.

When This Audit Is Worth the Effort

Worth it: any setup where training and execution information differ, any policy you intend to deploy, any result you plan to report. Also worth it for shared-parameter agents, where one agent's leak silently contaminates all of them because they share weights.

Overkill: fully centralized execution, where the same information is available at both times. Also overkill for pure research ablations where you control the information contract by construction and the audit would only confirm what you already asserted.

The cost is real: the guard and logging add friction to every forward pass. Keep them behind a flag for long runs, and enable them for the first few thousand steps and any time you change the actor's input wiring.

Extend the Experiment

One meaningful modification turns this audit into a habit.

Swap the shared team reward for per-agent rewards and re-run. The audit still holds, but the coordination dynamics change, and you may find that agents specialize differently. Then add a second agent type with a different observation space and confirm the guard catches shape or provenance mismatches across heterogeneous inputs.

The larger move: replace the hand-rolled learner with a standard centralized-critic implementation and run the same harness against it. Record the audit log as a reusable artifact so future experiments inherit the check. The provenance-guard pattern transfers directly to communication-based setups, where messages become the new execution-time input and the same question applies: does the actor depend on a message that will not arrive?

Before you trust any CTDE result, prove that the execution path touches only execution-available information. Treat the audit log as a first-class artifact, not a debugging afterthought. The curve tells you it learned. The log tells you what it learned from.