Compare Shared and Counterfactual Credit Signals in a Toy Team Task

Two agents, one shared reward, and a number that refuses to say who earned it. The fastest way to understand that refusal is to force it into arithmetic: build a task small enough to enumerate, compute the shared signal, compute a difference-style signal, and watch the two disagree in a way that teaches you something instead of breaking.

By the end of this exercise you will be able to name exactly what a counterfactual holds fixed, and why that choice — not the agent's skill — determines the attribution you get.

This assumes you already know why shared outcomes create an attribution problem. Here we only need the task definition and the enumeration.

The Task We Will Compute On

We want a cooperative task small enough to enumerate every joint action in a table. Two agents, a small joint action set, and a shared scalar reward that depends on the joint action. That is the whole instrument.

Why a finite one-step task? Because it isolates structural credit assignment — which agent caused the outcome — from temporal credit assignment and from value-function approximation error. If you add time steps or a neural critic, you cannot tell whether a weird signal came from the attribution method or from the estimator. One step, exact enumeration, no sampling noise.

Here is the task. Two agents each choose wait or go. The team gets a shared reward of 1 only when exactly one agent goes; if both go or both wait, the reward is 0. This is the classic "pass in order" structure, and it has two optimal joint actions: (go, wait) and (wait, go).

import itertools

ACTIONS = ["wait", "go"]
AGENTS = ["a1", "a2"]

def joint_reward(joint_action):
    # joint_action is a tuple aligned with AGENTS
    goes = sum(1 for a in joint_action if a == "go")
    return 1.0 if goes == 1 else 0.0

# Enumerate every joint action and print the table.
joint_actions = list(itertools.product(ACTIONS, repeat=len(AGENTS)))
for ja in joint_actions:
    print(dict(zip(AGENTS, ja)), "->", joint_reward(ja))

Expected output:

{'a1': 'wait', 'a2': 'wait'} -> 0.0
{'a1': 'wait', 'a2': 'go'}   -> 1.0
{'a1': 'go',   'a2': 'wait'} -> 1.0
{'a1': 'go',   'a2': 'go'}   -> 0.0

Four joint actions, two of them optimal. Print this table before computing anything else. Every signal below is a function of these four numbers.

Shared Signal: One Number for the Whole Team

The shared signal is the expected return under the current joint policy. Give each agent a policy over {wait, go}, enumerate joint actions, and weight each by its probability.

policy = {
    "a1": {"wait": 0.5, "go": 0.5},
    "a2": {"wait": 0.5, "go": 0.5},
}

def joint_prob(ja):
    p = 1.0
    for agent, action in zip(AGENTS, ja):
        p *= policy[agent][action]
    return p

shared_signal = sum(joint_prob(ja) * joint_reward(ja) for ja in joint_actions)
print("shared signal per agent:", shared_signal, shared_signal)

Expected output:

shared signal per agent: 0.5 0.5

That is the whole point. The shared signal is a property of the team, not the agent. Agent 1 and agent 2 receive the same number — 0.5 — regardless of what either did. An agent that contributed nothing and an agent that carried the task get the identical learning signal.

Success criterion for this milestone: a single scalar per agent that is provably equal across agents. If your two numbers differ, you have a bug in the enumeration, not a discovery.

Counterfactual Signal: Replace One Agent, Hold the Rest Fixed

Now the difference-style signal. The counterfactual for agent i replaces i's action with a reference action while every other agent's action and the environment state stay at their factual values. Then we measure how the reward changes.

The held-fixed set is the entire lesson. When you compute agent i's counterfactual, you hold fixed:

• the other agents' actions,
• the state,
• the transition,
• the reward function.

Change any one of those and the attribution changes. Write that list down before you write the code.

def counterfactual_contribution(agent, joint_action, reference_action):
    # Factual reward under the observed joint action.
    factual = joint_reward(joint_action)
    # Counterfactual: swap only this agent's action, hold the rest fixed.
    cf_action = tuple(
        reference_action if a == agent else act
        for a, act in zip(AGENTS, joint_action)
    )
    return factual - joint_reward(cf_action)

# Reference choice A: a fixed baseline action.
reference = "wait"
print("reference = fixed 'wait'")
for ja in joint_actions:
    for agent in AGENTS:
        c = counterfactual_contribution(agent, ja, reference)
        print(dict(zip(AGENTS, ja)), agent, "->", c)

Expected output (abbreviated):

reference = fixed 'wait'
{'a1': 'wait', 'a2': 'go'}   a1 -> 0.0
{'a1': 'wait', 'a2': 'go'}   a2 -> 1.0
{'a1': 'go',   'a2': 'wait'} a1 -> 1.0
{'a1': 'go',   'a2': 'wait'} a2 -> 0.0

Read the first two lines. When the factual joint action is (wait, go), agent 2's contribution is 1.0 and agent 1's is 0.0. The signal separates them. Now switch the reference to the agent's own policy — marginalize over what the agent would have done anyway:

def policy_marginal_contribution(agent, joint_action):
    # Average the counterfactual over the agent's own action distribution.
    total = 0.0
    for ref in ACTIONS:
        total += policy[agent][ref] * counterfactual_contribution(agent, joint_action, ref)
    return total

print("reference = expectation over own policy")
for ja in joint_actions:
    for agent in AGENTS:
        print(dict(zip(AGENTS, ja)), agent, "->", policy_marginal_contribution(agent, ja))

Expected output:

reference = expectation over own policy
{'a1': 'wait', 'a2': 'wait'} a1 -> 0.0
{'a1': 'wait', 'a2': 'wait'} a2 -> 0.0
{'a1': 'wait', 'a2': 'go'}   a1 -> 0.5
{'a1': 'wait', 'a2': 'go'}   a2 -> 0.5
{'a1': 'go',   'a2': 'wait'} a1 -> 0.5
{'a1': 'go',   'a2': 'wait'} a2 -> 0.5
{'a1': 'go',   'a2': 'go'}   a1 -> 0.0
{'a1': 'go',   'a2': 'go'}   a2 -> 0.0

The numbers move. Under a uniform policy, the fixed-baseline signal and the policy-marginal signal differ because the reference distribution weights the two counterfactual outcomes differently. The attribution is reference-dependent. It is not a property of the agent; it is a property of the agent under a chosen baseline.

Notice the degeneracy in the output. At the two optimal joint actions, (wait, go) and (go, wait), both agents get 0.5. At the two failing joint actions, (wait, wait) and (go, go), both agents get 0.0. The policy-marginal signal cannot distinguish the agent who acted correctly from the agent who got lucky — because under a uniform policy, the reference distribution is symmetric. This is not a bug. It is the baseline doing exactly what you told it to do.

Comparing the Two Signals Side by Side

Two cases worth running by hand.

Case 1 — shared ties, counterfactual separates. At joint action (wait, go), the shared signal gives both agents 0.5. The counterfactual with a wait reference gives agent 2 1.0 and agent 1 0.0. Same team outcome, different attribution.

Case 2 — both uninformative. At (go, go), the reward is 0. Swap either agent's action to wait and you land on an optimal joint action, so the counterfactual contribution is 1.0 for both — but the team is failing. The signal tells you "change something," not "who is right." At (wait, wait), the same swap logic gives 0.0 for both, because the reference action wait is already what the agent did. The signal is silent, not because the agent is blameless, but because the baseline is a no-op.

There is a known degeneracy here. When multiple joint actions are equally optimal, a difference-style signal can average to zero for every agent even though the team is doing well. This is not a broken calculation. The task has multiple optima, and a baseline that swaps one agent at a time cannot tell which optimum the team is heading toward. The signal is honest about the ambiguity; the ambiguity is real.

Decision boundary: use the shared signal when you only need team-level learning. Use a counterfactual signal when you need per-agent gradients and can afford the extra computation plus the reference-action choice.

Failure Modes and Debugging Signals

All-zero counterfactual contributions. Check whether the task has multiple equivalent optima, or whether the reference action was chosen so the factual and counterfactual rewards coincide. In our task, a reference of wait at joint action (wait, wait) gives both agents 0.0 — not because neither agent did anything wrong, but because swapping wait for wait is a no-op.

Sign flips when the reference changes. This is expected. The attribution is reference-dependent. Report the reference explicitly whenever you report a contribution number, or the number is meaningless.

Confusing structural with temporal credit assignment. This exercise holds time fixed. A long-horizon task adds a second, separate attribution problem: which timestep mattered. Do not debug one by staring at the other.

Over-reading a single number. A contribution signal is an estimate under a chosen baseline, not a causal verdict about the agent's skill. Treat it as a gradient hint, not a scorecard.

One Modification Worth Running

Change the reference action and predict the result before you run it. Switch from the fixed wait baseline to the policy-marginal version and ask: which agents' contributions move, and in which direction? Write your prediction down, then run the code. The gap between prediction and output is where the learning is.

Then add a third agent. The held-fixed set grows from one other action to two, and the counterfactual becomes more expensive to compute exactly — you now enumerate |A|^3 joint actions instead of |A|^2. This is the first hint of why exact counterfactuals do not scale.

Optional: replace exact enumeration with a sampled estimate. Draw joint actions from the policy, compute the contribution on each sample, and measure how much the signal varies across seeds. That variance is the reason these signals are noisy in real multi-agent RL, and it is the bridge to the next step.

The shared signal tells you the team is doing well. The counterfactual signal tells you who to nudge. The counterfactual is only as meaningful as the baseline you chose — and the moment you move to a sampled, multi-step setting, the assumption that will break first is the one you have been leaning on all along: that you can hold everything else fixed while you change one agent.