Deriving the Expected SARSA Target With a Worked Action-Value Example

You know the SARSA target. You know the Q-learning target. Then someone says "Expected SARSA" and your brain stalls on one word: expectation. What action is it expecting?

None. That is the whole trick. Expected SARSA does not wait to see which action gets sampled next. It averages over all of them, weighted by how likely the policy is to pick each one. Once that clicks, the arithmetic is ordinary multiplication and addition.

Here is the full derivation, then a worked number you can check by hand.

Where the Expected SARSA Target Comes From

You already have the two targets this article builds on:

• SARSA: $R + \gamma Q(S', A')$, where $A'$ is the action the policy actually sampled.
• Q-learning: $R + \gamma \max_{a'} Q(S', a')$, the best estimated next action.

SARSA's target inherits the randomness of $A'$. If the policy is stochastic, the same state-action pair can produce a different target on every visit, even when the reward and next state are identical. That variance slows learning and makes updates jumpy.

Expected SARSA removes that source of noise. Instead of waiting to observe $A'$, it computes the policy-weighted average of every next-state action value:

$R + \gamma \sum_{a'} \pi(a' \mid S')\, Q(S', a')$

The expectation is taken with respect to the target policy $\pi$ — the policy you are learning about — not the environment and not the behavior policy that generated the data. That distinction matters, and it is exactly what makes Expected SARSA's on-policy or off-policy status a design choice rather than a fixed label.

Notation and Assumptions Before the Derivation

Pin the symbols down first so the derivation reads as arithmetic.

• $S, A$: current state and action.
• $R$: reward received after taking $A$ in $S$.
• $S'$: the next state.
• $a'$: a candidate next action, ranging over all actions available in $S'$.
• $\gamma$: discount factor.
• $\alpha$: step size.
• $Q(s, a)$: the tabular action-value estimate.
• $\pi(a' \mid S')$: the target policy's probability of choosing $a'$ in $S'$.

The assumptions are the usual tabular ones: finite state and action spaces, a table of $Q$ values, a one-step bootstrap, and a policy $\pi$ whose probabilities over next actions you can evaluate. That last assumption is the load-bearing one. If $\pi$ is unknown or you cannot compute its probabilities, you cannot form the expectation. If the action space is continuous, the sum becomes an intractable integral.

Deriving the Target Step by Step

Start from the general one-step backup. The target is the immediate reward plus the discounted value of the next state under the policy:

$\text{target} = R + \gamma\, V(S')$

Now expand $V(S')$ as the policy-weighted sum of action values in that state:

$V(S') = \sum_{a'} \pi(a' \mid S')\, Q(S', a')$

Substitute, and you have the Expected SARSA target:

$\text{target} = R + \gamma \sum_{a'} \pi(a' \mid S')\, Q(S', a')$

The update applies the TD error — the bracket, not the target itself:

$Q(S, A) \leftarrow Q(S, A) + \alpha \Big[\, R + \gamma \sum_{a'} \pi(a' \mid S')\, Q(S', a') - Q(S, A) \,\Big]$

The bracket is the correction signal. It says: here is how far my current estimate was from a better one-step guess. The target is the guess; the TD error is the gap.

Two boundary cases fall out of the same formula:

• If $\pi$ is greedy, all probability mass sits on the argmax, the sum collapses to $\max_{a'} Q(S', a')$, and you recover Q-learning.
• If $\pi$ puts all mass on a single action, you recover SARSA's sampled form.

Q-learning is not a separate species. It is Expected SARSA with a greedy target policy.

Picture the backup as a small tree: from $S$ you take $A$, receive $R$ and land in $S'$, then branch over every $a'$ — each branch weighted by $\pi(a' \mid S')$, each leaf holding $Q(S', a')$. The expectation is the weighted average across those leaves.

Worked Example: Computing the Target by Hand

Set up a tiny case. From state $S$ you take action $A$, receive $R = 2$, and land in $S'$. The discount is $\gamma = 0.9$. In $S'$ there are three actions with these values:

Case 1: Uniform policy

Suppose $\pi(a' \mid S') = 1/3$ for each action. Compute the weighted sum term by term:

The sum is $1.333 + 2.667 + 2.000 = 6.0$. The target is:

$2 + 0.9 \times 6.0 = 2 + 5.4 = 7.4$

Now apply the update. Suppose $Q(S, A) = 5$ and $\alpha = 0.1$. The TD error is $7.4 - 5 = 2.4$, so the new value is $5 + 0.1 \times 2.4 = 5.24$.

Case 2: ε-greedy policy

Let $\epsilon = 0.2$. The greedy action is $a_2$ (value 8). Under ε-greedy, $a_2$ gets $1 - \epsilon + \epsilon/3 = 0.8 + 0.0667 = 0.8667$, and the other two get $\epsilon/3 = 0.0667$ each.

The sum is $0.267 + 6.933 + 0.400 = 7.6$. The target is $2 + 0.9 \times 7.6 = 8.84$. The TD error against $Q(S, A) = 5$ is $3.84$, and the new value is $5 + 0.1 \times 3.84 = 5.384$.

Case 3: Greedy policy

Set $\pi(a_2 \mid S') = 1$ and the rest to zero. The sum is just $Q(S', a_2) = 8$. The target is $2 + 0.9 \times 8 = 9.2$ — identical to the Q-learning target. Same formula, different weights.

Common mistake: Treating the expectation as a fourth action or a special case. It is a weighted average. If you can write down $\pi$ and read off $Q$, you can compute it with a loop and a running sum.

Policy Dependence: On-Policy, Off-Policy, or Both

The confusion around Expected SARSA usually comes from asking whether it is on-policy. The honest answer: it depends on which policy supplies the expectation, and you choose that.

The variance column needs a careful reading. It describes only the randomness that comes from which next action gets selected. SARSA samples one action, so its target moves with that draw. Expected SARSA and Q-learning both replace the draw with a deterministic function of the current $Q$ values — a weighted average and a max, respectively — so neither inherits action-sampling noise on a fixed transition.

That is not the same as saying their whole updates are low-variance. The reward $R$ and next state $S'$ are still sampled from the environment, and the $Q$ estimates themselves carry noise. If the environment is stochastic, the target can still swing from step to step. The precise claim is narrower: the expectation removes the variance contributed by next-action sampling, conditional on the same transition and the same value estimates. Whether that translates into smoother learning curves depends on how much of the total noise came from action sampling in the first place.

If $\pi$ equals the behavior policy, the update is on-policy and matches SARSA's intent. If $\pi$ is greedy with respect to $Q$, you get Q-learning. If $\pi$ is some other target policy, you can compute the expectation off-policy without importance sampling on the final step — a practical advantage over sampled SARSA, which needs importance ratios to correct for the mismatch.

That flexibility is the point. Expected SARSA is not "between" SARSA and Q-learning on some spectrum. It is a generalization that contains both as special cases.

When the Expectation Pays Off

SARSA's target varies with the sampled $A'$. Expected SARSA removes that specific variance by averaging over all next actions. The trade is computation: you must evaluate every next action at every update.

In a table with a handful of actions, that cost is trivial. In a continuous action space, it is a different story — you cannot sum over infinitely many actions, so you fall back to sampling.

Where the expectation helps:

• Stochastic policies where the sampled action swings the target wildly.
• Settings where stable learning matters more than raw update speed.
• Off-policy learning where you want to avoid importance sampling on the last step.

Where it does not:

• Very large or continuous action spaces.
• Unknown target policies you cannot evaluate.
• Cases where the extra computation per update outweighs the variance reduction.

Common mistake: Assuming lower variance means lower bias. The expectation is still a bootstrapped estimate, so it inherits the bias of $Q$. Variance reduction and bias reduction are different problems.

This expectation-over-actions idea reappears in modern actor-critic methods, where the policy is a distribution and the value target averages over sampled or reparameterized actions. The tabular version is the cleanest place to see the mechanism before the policy becomes a neural network.

What to Do Next

If you can enumerate next actions and you know the policy, use the expectation. If the action space is continuous or the policy is unknown, fall back to sampling.

The concrete next step: implement the Expected SARSA update on a small tabular environment — a gridworld with four actions is enough — and log the TD error at each step. Run it alongside SARSA and compare the variance of the updates. You will see the difference in the numbers, not just in the formula. From there, the same expectation appears in policy-gradient and actor-critic targets, where the policy is a distribution you sample from rather than a table you read.