Why Policy-Gradient Baselines Reduce Variance Without Changing the Gradient

Two runs of the same policy-gradient code. Same environment, same seed, same learning rate. One climbs. One flatlines. The usual advice is "subtract a baseline," and it works often enough that most of us accept it as a trick. But a trick you cannot derive is a trick you cannot debug.

Here is the sharper model: a baseline is a control variate. Its expected contribution to the gradient is exactly zero, which means it has no authority over the direction of learning. It can only move the noise. Once you see that, the design question changes from "should I use a baseline?" to "which baseline centers my weights without adding its own error?"

Let's derive it.

The Baseline Is a Control Variate, Not a Correction

You already know the sampled policy-gradient estimator. For a trajectory of states, actions, and returns, the update is a sum of score-function terms, each weighted by a return-like quantity:

$\hat{g} = \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \Phi_t$

where $\Phi_t$ is the weight — a return, a reward-to-go, or a value estimate. The score function $\nabla_\theta \log \pi_\theta(a_t \mid s_t)$ tells you which direction to push the action probability. The weight tells you how hard to push.

A baseline is a function $b(s_t)$ subtracted from that weight:

$\hat{g}_b = \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \bigl(\Phi_t - b(s_t)\bigr)$

Two constraints define it. The baseline depends on the state only — not on the action taken, not on the policy parameters. And the expectation is taken under the same policy $\pi_\theta$ that generated the samples. Everything downstream depends on those two conditions.

Why call it a control variate? In Monte Carlo estimation, a control variate is a quantity you subtract from a noisy estimator because you know its expected value. If that expected value is zero, you have removed noise without moving the target. That is precisely what an action-independent baseline does.

Note: This article assumes the likelihood-ratio derivation — the log-derivative identity that turns the expected-return objective into a score-function estimator. If that step is not yet solid, the algebra below will look like sleight of hand. The mechanism is what matters here, not a re-derivation.

Why an Action-Independent Baseline Leaves the Gradient Unchanged

Write the expected contribution of the subtracted term. Under the state-visitation distribution $d(s)$ and the policy, it is:

$\mathbb{E}\bigl[b(s_t)\nabla_\theta \log \pi_\theta(a_t \mid s_t)\bigr] = \sum_s d(s)\, b(s) \sum_a \pi_\theta(a \mid s)\, \nabla_\theta \log \pi_\theta(a \mid s)$

Now apply the log-derivative identity, $\nabla_\theta \log \pi_\theta(a \mid s) = \dfrac{\nabla_\theta \pi_\theta(a \mid s)}{\pi_\theta(a \mid s)}$. The policy factor cancels:

$= \sum_s d(s)\, b(s) \sum_a \nabla_\theta \pi_\theta(a \mid s)$

The inner sum is the gradient of the action probabilities over all actions. Those probabilities must sum to 1 for every state, and the gradient of a constant is zero:

$= \sum_s d(s)\, b(s)\, \nabla_\theta \underbrace{\sum_a \pi_\theta(a \mid s)}_{=\,1} = 0$

That is the whole result. The subtracted term vanishes in expectation for any $b(s)$. The baseline is free. It cannot bias the gradient, which is exactly why you are allowed to choose it for variance rather than for correctness.

Now watch the assumption do its work. If the baseline depended on the action — $b(s, a)$ — the subtracted expectation becomes:

$\mathbb{E}\bigl[b(s_t, a_t)\nabla_\theta \log \pi_\theta(a_t \mid s_t)\bigr] = \sum_s d(s) \sum_a b(s,a)\, \nabla_\theta \pi_\theta(a \mid s)$

The $b(s,a)$ factor sits inside the action sum, so it cannot be pulled out front. The inner sum is no longer $\nabla_\theta \sum_a \pi_\theta(a \mid s)$; it is a weighted sum of per-action probability gradients, and that weighted sum generally does not vanish. Action-independence is not a stylistic preference. It is the load-bearing wall.

Common mistake: Assuming every action-dependent baseline is automatically biased. Some action-dependent estimators add a correction term that restores unbiasedness. The clean invariance result belongs to the state-only case; anything else needs its own proof.

Where the Variance Actually Lives

The estimator's variance comes from two sources: the spread of the score function, and the spread of the weight multiplying it. The baseline touches only the second.

Consider raw returns as the weight. The same action in the same state can be followed by a great return in one episode and a poor one in the next, because everything after that action is noisy. The weight swings wildly even when the action's merit has not changed. That swing is the variance you feel as unstable learning.

Subtracting $b(s_t)$ shifts the weight's center. It does not touch the score function at all. So it attacks the spread directly: instead of weights scattered around some large absolute return, you get weights scattered around zero, with a much smaller range.

Here is the honest limit. The variance-minimizing baseline is not simply the value function. The exact optimum weights rewards by the squared norm of the score function:

$b^* = \frac{\mathbb{E}\bigl[\Phi_t\, \|\nabla_\theta \log \pi_\theta(a_t \mid s_t)\|^2\bigr]}{\mathbb{E}\bigl[\|\nabla_\theta \log \pi_\theta(a_t \mid s_t)\|^2\bigr]}$

The value function is a practical approximation to this, not the exact answer. Keep that in mind when someone claims $V(s)$ is "the optimal baseline." It is the convenient one.

Why the State-Value Function Is the Practical Baseline

The state-value function $V^\pi(s)$ is the expected return from state $s$ under the current policy. By construction it depends on the state alone — so it satisfies the invariance condition for free.

Substituting it turns the weight into the advantage:

$A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)$

Positive advantage means the action did better than the state's average. Negative means worse. The update now compares an action to its own context instead of to an absolute number. An agent that gets exactly what it expected should feel neutral — and the baseline is what encodes that neutrality.

A Small Worked Example

One state, two actions, four sampled returns. Say the policy currently assigns $0.5$ to each action, and the score function magnitude is the same for both, so we can compare weights directly.

Without a baseline, the weights are $\{10, 2, 8, 0\}$ — mean 5, spread 10. With $b = 5$, they are $\{+5, -3, +3, -5\}$ — mean 0, spread 10 in range but centered, and the sign now carries meaning: action A was reinforced on sample 1 and suppressed on sample 2.

The gradient direction is unchanged in expectation. What changed is that the weight now tells the policy relative merit. Smaller, better-centered weights mean fewer samples to average out the noise.

Warning: This example illustrates recentering and sign interpretation, not variance reduction. Subtracting the same constant from every return leaves the variance of the weights unchanged in this equal-score, single-state setup. Variance reduction appears when the baseline correlates with the score-weighted return across states — the case the next section addresses.

What Approximation Error Does to the Benefit

In practice you do not have $V^\pi(s)$. You have a learned critic $\hat{v}(s; w)$ that is wrong in two distinct ways.

First, its own estimation error. Second, the fact that it is evaluated under a policy that keeps changing. The critic chases a value function the actor keeps invalidating, so the baseline is always slightly stale.

Here is the key consequence: a biased baseline does not bias the gradient in expectation. Invariance holds for any $b(s)$, good or bad. But a bad baseline can shrink the variance benefit, or reverse it. If the baseline's error is large relative to the return spread, you are subtracting something closer to noise than to a center, and the update noise returns.

The practical diagnostic: watch whether the critic's predictions track returns. If they do not, the baseline is a random shift wearing a value function's clothes.

Common mistake: Treating a baseline as a correctness fix. It is not. A zero baseline is unbiased but noisy. A perfect baseline is unavailable. The useful region is a baseline good enough to center the weights without adding its own noise.

When a Baseline Helps and When It Misleads

Use a baseline when returns are large and variable relative to the differences between actions. That is where centering buys the most.

Be cautious when the critic is trained on very little data, when the reward scale is unstable, or when the policy is changing faster than the value function can track.

One more correction, and it matters. A baseline does not improve the gradient's direction. It changes the noise around a direction that was already correct in expectation. And lower variance is not a complete explanation of why one baseline beats another — baselines with identical variance can produce different learning behavior, because they change which regions of the policy space the optimization visits. Variance is the first-order story, not the whole story.

The Decision Rule

The baseline is safe because its expected contribution is zero. It is useful because it centers the weight. It is limited because it is learned.

That is the entire derivation in three clauses. Now prove it to yourself. Instrument your own policy-gradient loop to log the spread of the weight term with and without a value baseline. Watch the estimator's noise change while the average update direction does not. That experiment is the proof that the derivation was not decoration.