Actor-Critic Methods Explained: Why Policies Need a Baseline

If you've worked through policy gradients, you've felt the problem: one lucky episode can make a mediocre action look brilliant, and the next update swings the policy in a direction it doesn't deserve. The temptation is to blame the reward signal. But the reward isn't wrong—it's noisy. The fix isn't a better grade. It's a baseline that subtracts what your policy already expects.

That baseline is the critic. And the architecture that pairs it with your policy—the actor—is the workhorse behind a major family of deep reinforcement learning methods.

The Problem Policy Gradients Leave Behind

Policy gradient methods have a clean idea at their core: push up the probability of actions that led to good returns, push down the ones that didn't. The update rule samples an episode, looks at the total return, and adjusts the policy accordingly.

The trouble is that returns are noisy. Imagine you're learning to play chess and you evaluate every move by playing the rest of the game once. A mediocre move that happens to lead to a blunder by your opponent looks excellent. A strong move that gets punished by a lucky counter looks terrible. Your gradient update doesn't know the difference between "this action was good" and "this action got lucky."

This is the variance problem. Each sampled return is a single, noisy estimate of how good an action really is. Average over enough samples and the noise cancels out—but "enough samples" can be painfully many. In deep reinforcement learning, where every environment interaction costs time and compute, raw policy gradients train slowly and erratically.

The real question isn't how to get a better reward signal. It's how to grade an action against what the policy already expects from that state, instead of against raw luck.

Why a Baseline Fixes Noise Without Changing the Goal

Here's the surprising part: you can subtract any state-dependent constant from the return, and the expected gradient doesn't change. The policy still moves toward genuinely better actions on average. What changes is the variance of each individual update.

Think about grading that chess move again. Instead of asking "how did the rest of the game go?", ask "how much better did this move do than what this policy usually achieves from this exact board position?" A move that leads to a win from a position where the policy usually wins anyway isn't informative. A move that leads to a win from a position where the policy usually loses is very informative.

That second question is the advantage: how much better was this action than the policy's average from this state?

The natural baseline is the state-value function, V(s)—the expected return from state s under the current policy. We don't know it in advance, so we learn it. And that learned value function is the critic.

The tradeoff is worth naming: the critic is learned, so it's imperfect. But a slightly wrong baseline beats raw noisy returns. The variance reduction you get from subtracting even an approximate expectation is usually substantial compared to the bias you introduce.

The Actor and the Critic: Two Jobs, One Loop

Actor-critic reinforcement learning splits the learning problem into two roles that stay stubbornly separate.

The actor is your policy network. It takes a state and outputs action probabilities (for discrete actions) or the parameters of a distribution over actions (for continuous control). Its job is to decide what to do.

The critic is your value network. It takes a state and outputs an estimate of expected return. Its job is to score how good a state is under the current policy. It never chooses actions. It only grades them.

That boundary matters more than it looks. In a continuous action space, you cannot turn a value function into a policy by scanning over actions—there are infinitely many. The critic is structurally incapable of replacing the actor. It exists to make the actor's learning signal cleaner, not to make the actor unnecessary.

Note: This article follows the most common setup, where the critic estimates the state-value function V(s). Some actor-critic variants use a critic that estimates Q(s,a) instead, which evaluates a specific action rather than a state. The core division of labor stays the same: the actor proposes actions, and the critic scores them.

The learning loop looks like this:

1. The actor observes a state and samples an action.
2. The environment returns a reward and the next state.
3. The critic updates toward a return target, improving its estimate of V(s).
4. The actor updates using the critic's feedback as its baseline.

Both networks learn simultaneously, from the same stream of experience. The critic gets better at predicting what the policy will achieve. The actor gets better at choosing actions that beat that prediction.

From Baseline to Advantage: Grading the Action, Not the State

Subtracting V(s) from the return gives you an estimate of the advantage: a measure of how much better a specific action was than what the policy usually achieves from that state.

A positive advantage means the action beat the policy's average. A negative advantage means it fell short. The policy gradient then becomes: increase the probability of actions with positive advantage, decrease the probability of actions with negative advantage.

There's a practical shortcut here. You don't need to learn both Q(s,a) and V(s) to estimate advantage. You can use a one-step return: the reward you just received plus the discounted value of the next state, minus the value of the current state.

That quantity—reward plus discounted next-state value minus current-state value—is the TD error. It's an estimate of advantage, not the definition of it. The true advantage asks how much better an action was than the policy's average; the TD error estimates that quantity from a single step of experience. Early in training, when the critic's estimates are rough, the TD error is a noisy proxy. As the critic improves, it becomes a sharper signal.

The TD error is the signal that updates both networks. The critic uses it to improve its value estimates. The actor uses it to decide whether the action it took was better or worse than average.

One dial worth understanding: how many steps of reward you include before bootstrapping from the critic's value estimate. A one-step TD error is simple and low-variance, but it leans heavily on the critic's estimate, which may be biased early in training. Longer n-step returns reduce that bias but reintroduce variance from the extra sampled rewards. This bias-variance dial is the single most useful mental model for navigating the actor-critic family.

Tracing One Update by Hand

The fastest way to make this concrete is to trace a single update on a tiny two-state example.

Imagine two states, A and B. Your policy currently picks action "left" from state A with probability 0.5. The critic estimates V(A) = 10. You take action "left," receive a reward of 12, and transition to state B, where the critic estimates V(B) = 0. Assume a discount factor of 1 for simplicity.

The one-step TD advantage estimate is 12 + 1 × 0 − 10 = 2. The action beat the critic's expectation by 2, so the actor increases the probability of "left" from state A.

Now imagine the same action, same reward, but the critic estimates V(A) = 15. The advantage estimate is 12 + 1 × 0 − 15 = −3. The action underperformed the critic's expectation, so the actor decreases the probability of "left."

Same action. Same reward. Opposite gradient direction. The critic's estimate didn't change what happened—it changed what the policy should learn from what happened. That's the entire job of the critic: to tell the actor whether an action was genuinely good or just lucky relative to what the policy already achieves.

Where Actor-Critic Methods Go Wrong

The most common failure mode is the moving-target problem. The actor and critic are chasing each other. The critic tries to predict what the actor will achieve. The actor tries to improve against the critic's predictions. If their learning rates are mismatched, both can drift—the critic chasing a policy that keeps changing, the actor chasing a baseline that keeps moving.

My rule: tune the critic's learning rate and update cadence separately from the actor's. They are not one network with two heads. They are two learners with different jobs, and they need different settings.

A second failure mode is an overconfident critic. If the value network becomes too sure of itself—especially early in training, when its estimates are based on limited experience—it feeds the actor misleading advantages. A critic that thinks a state is terrible when it's actually fine will make the actor avoid actions that were genuinely good.

Common mistake: Don't just watch the episode-return curve. Watch the critic itself. If predicted V(s) consistently overshoots or undershoots the returns the agent actually collects, the critic is biased, and every advantage estimate inherits that bias. If advantage magnitudes explode or collapse over time, the critic and actor are likely fighting each other rather than converging.

There's also the shared-network temptation. Sharing layers between actor and critic saves compute and can work well in practice. But it couples their gradients. When the critic updates, it changes the features the actor depends on, and vice versa. This coupling can destabilize training in ways that are hard to debug. If you're starting out, keep the networks separate until you understand why you need to share.

When to Reach for Actor-Critic (and When Not To)

Actor-critic methods are not automatically the best choice for every problem. They add a second network, a second learning rate, and a new set of tuning headaches. Use them when the problem demands it.

Reach for actor-critic when you need a directly parameterized policy: continuous action spaces, or problems where you want a stochastic policy that can explore meaningfully. Pure policy gradients work here too, but their variance makes training painfully slow. The critic's baseline is what makes direct policy learning practical.

Prefer pure value-based methods when the action space is small and discrete and you can maximize over actions cheaply. Q-learning and its relatives are often more sample-efficient in this setting because they can reuse past experience through replay buffers, and they don't require the actor-critic tuning burden.

The modern algorithms you've heard of—A2C, PPO, SAC, DDPG—are not separate inventions. They're different settings of the same actor-critic knobs. A2C is the cleanest expression of the advantage actor-critic idea. PPO adds a clipped objective that keeps the actor from changing too fast in a single update. SAC adds entropy regularization for better exploration. DDPG applies the actor-critic structure to deterministic policies in continuous control. Once you see the shared architecture, the variants stop looking like a zoo and start looking like a family.

The durable mental model: the critic is a learned, state-dependent baseline that tames noise without ever becoming the policy. It grades. The actor decides. Keep those jobs separate, tune them independently, and you'll be ready to understand why modern deep reinforcement learning methods are built on this foundation.

Your next step: take a policy-gradient implementation you already understand and add a value network that estimates V(s). Trace one update by hand, exactly as above, and verify that the actor's gradient direction flips when you change the critic's estimate. That experiment will teach you more about actor-critic methods than any diagram can.