SARSA vs Q-Learning: On-Policy and Off-Policy Learning Compared

Two agents start on the same grid. Same states, same actions, same rewards, same exploration schedule. One hugs the edge of a cliff and occasionally falls. The other keeps a cautious distance and rarely does. On paper, their update rules look nearly identical. In practice, they learn different policies because they trust different futures.

That single difference—whether the update assumes the best next action or the action the policy will actually take—is the whole SARSA vs Q-learning story. Get that decision rule straight, and everything else about these two algorithms follows.

Two Updates That Look Almost Identical

SARSA and Q-learning are both temporal-difference control methods. Both learn action values from experience, updating after each step using a reward plus a discounted estimate of the next state's value. Both handle exploration through policies like epsilon-greedy action selection.

Here is where they diverge:

Q-learning update: Q(s, a) ← Q(s, a) + α[r + γ maxₐ' Q(s', a') − Q(s, a)]

SARSA update: Q(s, a) ← Q(s, a) + α[r + γ Q(s', a') − Q(s, a)]

Look at the difference. In Q-learning, the term standing in for the next state's value is the maximum Q-value over all possible next actions. In SARSA, it is the Q-value of one specific action: the one the policy actually selects.

Everything else—learning rate, discount factor, the basic rhythm of observe, act, receive reward, update—is shared machinery. The question that separates these algorithms is about who the agent trusts in the future: the best action it could take, or the action it will actually take.

If you need a refresher on how the Q-learning update works step by step, the mechanics of the max-over-next-actions target are worth revisiting before you compare the two.

The One-Line Difference: Actual Action vs Best Action

The name SARSA is a mnemonic for the tuple that feeds its update: State, Action, Reward, State, Action. The algorithm needs the next action before it can learn from the current one, because that next action appears in the update target.

Walk through one transition for each algorithm and you will see exactly where the next action comes from.

SARSA: You are in state s, you pick action a using your epsilon-greedy policy, and you receive reward r, landing in state s'. Now you need to pick a' from s' using that same policy. Maybe it is the greedy action. Maybe epsilon fires and you explore with a random action. Either way, that actual a' is what enters the update. The value you back up is the value of the action your policy really takes.

Q-learning: You are in state s, you pick action a using your epsilon-greedy policy, and you receive reward r, landing in state s'. Now you look at all possible actions from s' and take the maximum Q-value. It does not matter whether your policy would have picked that action. The update assumes the best action follows.

That is the entire mechanical difference. Q-learning still needs a behavior policy to generate experience—it still explores, still takes random actions sometimes—but that behavior policy is decoupled from what the update assumes about the future. SARSA has no such decoupling. The action that generates experience and the action that appears in the update are the same action.

On-Policy vs Off-Policy: What the Labels Actually Mean

The terms on-policy and off-policy get misused constantly. They do not describe whether an agent explores. Both algorithms explore during training. The labels describe what the learned Q-function estimates.

SARSA is on-policy. It learns the value of the behavior policy itself, including all the exploration noise baked into that policy. If your behavior policy is epsilon-greedy with epsilon at 0.3, SARSA learns the value of following that specific policy—random actions and all. The learned Q-values answer the question: "If I follow my current policy from here, what return should I expect?"

Q-learning is off-policy. It learns the value of the optimal policy, assuming greedy action selection in the future. The update looks past the behavior policy toward the best possible policy, regardless of what the agent is actually doing during training.

Here is the practical consequence. SARSA's learned policy is only as good as the behavior policy it was trained under. If exploration is heavy, SARSA learns a policy that accounts for all those exploratory stumbles. Q-learning aims past the behavior policy toward the optimal one, even while the agent is still exploring.

A common confusion: off-policy does not mean the agent ignores its own actions. The behavior policy still generates all the experience. Q-learning just refuses to let that behavior policy dictate what the future is worth.

Why Q-Learning Can Be Riskier: The Cliff-Walking Case

The classic demonstration of this difference is cliff walking. Imagine a gridworld where the agent starts on the left, a goal waits on the right, and a cliff runs along the bottom edge. There are two sensible routes: a safe path along the top that is longer, and a shorter path that hugs the cliff. Step into the cliff and the episode ends with a large negative reward.

Run Q-learning on this grid and watch what happens. Near the cliff edge, the max-over-next-actions target creates optimism. When the agent is exploring, it sometimes steps toward the cliff. Q-learning updates the value of states near the edge as if the best action will follow next—even when the behavior policy is still random enough to stumble into disaster. The Q-values near the cliff get inflated because the update assumes a future the exploratory policy will not actually deliver.

SARSA updates from the action the policy actually takes. Near the cliff, that means the update sometimes includes the exploratory action that falls in. SARSA's Q-values near the edge reflect the real risk of the current policy. The result: SARSA learns a safer route that keeps distance from the cliff while epsilon is still high.

Here is the counterintuitive part. Q-learning converges to the optimal policy in the limit. Given enough visits to every state-action pair, it finds the greedy path that hugs the cliff safely. But during training, it takes more catastrophic falls. SARSA converges to a policy that is only near-optimal—it is optimal for its own exploratory behavior—yet it achieves higher average rewards during training because it avoids the cliff.

The cliff-walking case is not a special exception. It is the logical consequence of the one-line difference made visible. Q-learning is optimistic about risky states because it assumes the best action follows. SARSA is pessimistic about those same states because it assumes the policy will keep being itself.

When to Reach for Each Algorithm

The choice between SARSA and Q-learning comes down to one question: what does a bad exploratory step cost you?

Choose SARSA when failures during training are costly or dangerous. Physical robots, real-world control systems, any setting where an exploratory mistake has genuine consequences—SARSA's caution is worth the slower convergence toward optimality. The algorithm builds awareness of its own exploration into the values it learns, which keeps the agent away from states where exploration is expensive.

Choose Q-learning when training happens in simulation or when exploratory mistakes are cheap. Because Q-learning converges to the optimal policy more directly, it is the better default when you can afford to let the agent fall off cliffs during training. The off-policy structure also matters for a deeper reason: it is what later enables experience replay and learning from a separate behavior source. Off-policy methods can learn from data generated by any policy, which is why they dominate modern deep reinforcement learning. SARSA does not share that property.

One more note: decaying epsilon narrows the gap. As exploration shrinks, SARSA's behavior policy approaches greedy, and the policy SARSA learns approaches the optimal one. If you anneal epsilon aggressively, the safety difference between the two algorithms shrinks with it.

Common Mistakes When Comparing the Two

Mistake: assuming Q-learning does not explore. Both algorithms explore. The difference is what the update assumes about the future, not whether the agent takes random actions.

Mistake: thinking SARSA is always safer or Q-learning is always better. The safety difference is a training-time property. It matters most when epsilon is high and shrinks as exploration decays. In a low-risk environment with cheap resets, Q-learning's optimism is a feature, not a bug.

Mistake: confusing the behavior policy with the learned policy in Q-learning. Off-policy does not mean the agent's actions do not matter. The behavior policy generates the experience; the update just refuses to let that policy define the future's value.

Mistake: expecting both to converge to the same Q-values under the same exploration schedule. SARSA converges to the value of the behavior policy. Q-learning converges to the value of the optimal policy. Different targets, different destinations.

If you forget everything else, remember the decision rule: ask whether the update trusts the best next action or the action the policy will actually take. The answer tells you which algorithm you are looking at, what it learns, and where it will be cautious or bold.

The best way to make this stick is to run both. Build a small gridworld with a visible risk gradient—a cliff, a penalty zone, anything that punishes exploratory steps—and train both algorithms with the same epsilon schedule. Watch the learned paths diverge. Watch the episodic rewards tell different stories. Then decay epsilon and watch the gap narrow.

Two updates that look almost identical. One line of difference. A completely different path through the world.