Cooperative vs Competitive Multi-Agent RL: How Incentives Change Learning

"Cooperative" and "competitive" describe the reward wiring, not the environment—and the same physical scene can host either regime depending on how you connect the payoffs.

Picture two agents in a shared space. In one configuration, they both receive +1 when they occupy adjacent cells; in another, one receives +1 exactly when the other receives −1. The geometry is identical. The learning problem is not. Rewire the payoffs and you change what "optimal" means, what information matters, what failure looks like, and how you should evaluate the result.

This distinction is the first thing to get right in multi-agent reinforcement learning, because every later choice—algorithm family, training architecture, evaluation metric—inherits the assumptions you make here.

The Reward Function, Not the Environment, Sets the Game

The most common beginner mistake is labeling an environment "competitive" because agents look adversarial, or "cooperative" because they share a space. Behavior is downstream of incentives. The reward structure is upstream.

Three canonical regimes cover most multi-agent problems:

• Cooperative: agents share one reward signal. All agents optimize the same team return.
• Competitive: rewards are zero-sum. One agent's gain is exactly another's loss.
• Mixed-motive: rewards are partially aligned and partially opposed. Think team reward plus individual bonus, or asymmetric payoffs between groups.

The same environment can render all three ways. Take a pursuit task: two chasers and one evader in a bounded arena. Wire the chasers to share a reward for tagging the evader, and you have a cooperative problem between the chasers. Give the evader a mirrored penalty, and the chaser-evader interaction becomes competitive. Add a small individual bonus to whichever chaser makes contact, and the chasers now sit in a mixed-motive relationship with each other—still on the same team, but with a private incentive that can pull against coordination.

This is why "cooperative vs competitive multi-agent reinforcement learning" is not a question about your environment. It is a question about your multi-agent incentives: who gets which reward, and how those rewards relate.

But reward wiring is the first diagnostic, not the complete classification. Two problems can share identical reward alignment and still differ in observability, action constraints, or whether agents are teammates or opponents. Shared rewards do not guarantee easy coordination—agents may still disagree about which of several equally good joint behaviors to settle on. Zero-sum rewards do not automatically tell you whether a stable equilibrium is even the right operational target. Classify the payoffs first, then check the interaction structure before you commit to a method.

What Cooperation Changes: Shared Return and the Credit Problem

In a cooperative regime, every agent maximizes the same return. That sounds simpler, not harder. It is not.

The difficulty shifts from opposing incentives to a problem of attribution. When a team succeeds, the shared reward arrives as one number—but no single agent controlled the outcome. Which agent's action caused the success? Which agent's hesitation delayed it? The team return does not say. This is the credit-assignment problem, and it is the defining difficulty of team games reinforcement learning.

Cooperation also does not escape non-stationarity. Your teammates are learning too. Their policies shift as they train, which means the target you are optimizing against moves under you. The environment is not the only source of change; your own side is.

Two architectural patterns fit this regime especially well:

• Centralized training with decentralized execution (CTDE): agents train with access to global information—other agents' observations, actions, or both—but execute using only their local observations. Extra training-time information reduces coordination difficulty without requiring deployment-time communication.
• Value decomposition: the team's value function is factored into per-agent components, letting each agent choose actions from its local view while the sum remains consistent with team optimality.

Both approaches exist because the cooperative regime has a clear target: a joint policy that maximizes team return. The evaluation question is correspondingly clean—measure team performance, not individual reward. An agent that looks "bad" by individual reward may be doing exactly what the team needs.

One caveat: "team-optimal" does not mean "unconstrained." If agents execute with only local observations, the best decentralized joint policy may fall short of the best joint policy computed with full information. Value decomposition also rests on representational assumptions—the team value must actually factor into the components you choose. When those assumptions break, the clean target gets muddy.

What Competition Changes: Zero-Sum Structure and Opponent-Dependent Value

Competition replaces the credit problem with a different kind of difficulty: there is no single "best" policy.

In a zero-sum game, the value of a state depends on who you are facing. A policy that dominates one opponent may collapse against another. Optimality is not a property of the policy alone; it is a property of the policy relative to an opponent. This is the core conceptual shift from single-agent RL, where optimality is defined against a fixed environment.

Consider the simple_tag environment: chasers pursue evaders in a continuous 2D world with obstacles. Chasers receive +10 for touching an evader; the evader receives −10. The evader is faster. The chasers must coordinate to corner. Now ask: what is the best chasing policy? The answer depends entirely on the evader's policy. Against a naive evader, a simple chase works. Against an evader that feints and exploits overcommitment, the same policy gets punished.

This is why self-play and opponent modeling dominate competitive multi-agent reinforcement learning. Self-play replaces a fixed opponent with an evolving one, forcing the agent to continually adapt. Opponent modeling gives the agent a predictive handle on what the other learner is likely to do.

The failure mode is exploitability: a policy that beats the opponents it trained against but loses to anything else. A policy is not good because it wins against one adversary; it is good because it wins against a distribution of adversaries. Evaluation must reflect this. Win rate against a held-out set of opponents—not absolute return—is the metric that matters.

Two clarifications keep this section honest. First, self-play is a training and evaluation process, not an equilibrium concept. It is one strategy for exploring the space of opponent policies—useful when you expect adversaries to adapt, overkill when you face a fixed or simple opponent. Second, zero-sum structure alone does not tell you which opponent distribution matters. If your deployment setting involves a known opponent class, training against that class may beat generic self-play. Match the training population to the deployment reality.

Mixed Motives: Where Cooperation and Competition Coexist

Most real systems do not fit either pure regime. They are mixed-motive multi-agent RL problems: partially aligned, partially opposed.

The canonical example comes from the MADDPG paper's environments, where teams of chasers coordinate internally while competing against evaders. Within the chasing team, rewards are shared—cooperative. Between chasers and evaders, rewards are zero-sum—competitive. Both structures operate simultaneously, and neither pure-regime assumption holds.

Mixed-motive settings create two distinctive challenges.

First, information asymmetry becomes a design lever. In a pure cooperative regime, sharing information is usually unambiguously good. In a mixed setting, what each agent observes—and who shares rewards with whom—changes the strategic character of the problem. An agent that knows its teammate's intentions coordinates better; an agent that reveals its own intentions to an opponent becomes exploitable.

Second, parameter sharing breaks down across groups with different reward functions. Sharing parameters between agents works when agents share the same reward function, because their critics need to assign similar values to similar states. In mixed settings, agents in different groups must learn different values for the same state—one agent's gain is another's loss. A shared critic cannot represent both. The practical rule: share parameters within a group that shares rewards, and treat cross-group sharing as a design risk rather than a default. You can sometimes condition a shared representation on group identity or keep separate value heads, but you cannot expect one value function to hold opposing evaluations of the same state.

The evaluation problem is the hardest here. In pure cooperation, team return is the metric. In pure competition, win rate against an opponent distribution. In mixed settings, you must ask: which outcome are you actually measuring? Team performance? Individual reward? Opponent outcome? The answer depends on which relationship you care about, and it is easy to measure the wrong one.

A workable recipe: report the relevant group and individual returns separately, include outcome or win measures for opposing groups, and test across opponent or teammate populations when adaptation matters. If you collapse everything into one number, you invite a policy that games that number at the expense of the relationship you actually care about.

A Comparison Table: Rewards, Information, Equilibrium, Evaluation

Axis	Cooperative	Competitive	Mixed-Motive
Reward structure	Shared team reward	Zero-sum: one agent's gain is another's loss	Partial alignment: team reward plus individual incentives, or asymmetric payoffs
Information assumptions	CTDE often useful: centralized training, decentralized execution	Opponent-dependent; self-play and opponent modeling are strategies, not automatic defaults	Information asymmetry is a design lever; sharing works within groups, not across opposed ones
Optimization target	Team-optimal joint policy, subject to execution constraints	No single best policy; value is opponent-relative	No single clear target; equilibrium selection is ambiguous
Evaluation metric	Team return	Win rate against a distribution of opponents	Report group and individual returns separately, plus outcome measures across groups
Typical failure mode	Credit-assignment failure: team cannot tell which agent caused the outcome	Exploitability: policy beats trained opponents but loses to others	Misaligned evaluation: measuring the wrong outcome for the question you care about

Note what the table does not say. "Equilibrium" is not a single concept that maps cleanly onto regimes. Team optimization asks whether the joint behavior maximizes the shared objective. Nash equilibrium asks whether any single agent could profit by changing its behavior while others stay fixed. Self-play is a training and evaluation process, not itself an equilibrium concept. Keep these separate in your head, or you will start treating algorithmic choices as logical consequences of the reward structure rather than contingent design decisions.

A Decision Rule for Classifying Your Problem

When you encounter a new multi-agent problem, do not guess from behavior. Run the classification explicitly:

1. Write down each agent's reward function. Do not assume from observed behavior. Write the actual reward expressions.
2. Check whether rewards sum to a constant. If one agent's reward is always the negative of another's, you have zero-sum competition. If all agents receive identical rewards, you have cooperation.
3. If neither, name the partial alignment explicitly. State which agents share rewards, which oppose each other, and where individual incentives diverge from team incentives.
4. Choose evaluation that matches the regime. Team return for cooperation. Win rate against an opponent distribution for competition. For mixed motives, report group and individual returns separately, include outcome measures for opposing groups, and test across populations when adaptation matters.
5. Select learning assumptions consistent with the regime. CTDE and value decomposition for cooperation when execution is decentralized. Self-play and opponent modeling for competition when opponents adapt. Careful grouping and information design for mixed settings.

A common mistake: labeling an environment "competitive" because agents look adversarial when rewards are actually shared. Two robots fighting over a single object might both receive the same reward for successful retrieval—the behavior looks competitive, but the incentive structure is cooperative. The learning problem follows the rewards, not the appearance.

When the classification is genuinely ambiguous, treat it as mixed-motive and state your assumptions openly. The danger is not ambiguity; it is silently importing assumptions from the wrong regime.

Turn Classification into a Habit

Take one multi-agent problem you already know—simple_tag, a cooperative navigation task, or a system you are actually building. Write down the reward functions. Classify it with the decision rule. Then state which evaluation metric and which learning assumptions the classification implies.

The taxonomy is not the point. The habit is: every time you encounter a multi-agent problem, the first question is not "what algorithm should I use?" It is "what are the incentives, and what does optimal even mean here?" Answer that, and the algorithm choice starts to look almost mechanical.