Prediction vs Control in Reinforcement Learning: Evaluating Policies and Improving Them

Scoring a policy and improving a policy feel like the same job. They are not. One is a measurement. The other is a search. Confusing them is one of the most common stalls for beginners who have just learned value functions — and it quietly blocks progress until the boundary becomes visible.

Here is the distinction in one sentence: prediction asks how good a fixed policy is, while control asks which policy you should use. Evaluation describes. Control decides. This article will make that boundary precise enough that you can name which problem you are solving before you run another experiment.

The Confusion: Knowing How Good a Policy Is vs Making It Better

When you first learn about value functions, a seductive thought appears: if I can compute how good a policy is, I can just pick the best one. The logic feels airtight. Score every policy. Choose the highest score. Done.

The flaw is hidden in the word compute. A value estimate tells you how much return a policy expects to collect. It does not tell you which action to change, in which state, to get more return. A measurement gives you a number. Improvement requires a decision.

Think about a commute. You can rate your current route: forty minutes on average, with occasional traffic spikes. That rating is useful. It tells you the route is tolerable or unbearable. But the rating alone does not tell you whether turning left at the third intersection instead of going straight will save you ten minutes. To find a faster route, you need to compare alternatives, form a hypothesis about which change helps, and test it.

Value estimation is the route rating. Policy improvement is the search for a better route. They support each other, but they are different jobs.

What Prediction Means: Scoring a Fixed Policy

The prediction problem in reinforcement learning is straightforward: estimate the expected return of a policy you are not changing.

If you have read about value functions, you already know the tools. The state-value function V(s) estimates the expected discounted return from state s when following a given policy. The action-value function Q(s, a) estimates the expected return from state s after taking action a, then following that same policy afterward. Both functions are defined with respect to a specific policy. Change the policy, and the values change with it.

The defining feature of prediction is that the policy stays frozen while you measure it. You are not asking "what should the agent do?" You are asking "if the agent follows this exact decision rule, how much return should it expect?"

Consider a tiny grid world where an agent always moves left, no matter what. That is a policy — a bad one, probably, but a policy. Prediction evaluates it. The agent follows the always-left rule, collects rewards along the way, and you estimate the expected return from each state. The output is a value estimate: "from the start state, this policy expects to collect roughly 2.3 total discounted reward."

That number matters even though the policy is bad. Prediction lets you compare two fixed strategies honestly. It lets you debug an environment by checking whether a sensible policy produces sensible values. It gives you a baseline before you try anything clever.

The key habit to build: when you compute a value estimate, ask yourself which policy produced this number? The value describes that policy and only that policy.

What Control Means: Searching for a Better Policy

The control problem asks a different question: which policy maximizes expected return?

Control is the actual goal of reinforcement learning. The agent should act well, not merely be scored well. That requires changing decisions — moving from "always go left" to "go left when the corridor is clear, go right when the reward is ahead."

Control needs two ingredients that prediction does not provide. First, it needs a way to compare alternatives: given the choice between action A and action B in this state, which leads to more return? Second, it needs a rule for choosing among them: once you know which action looks better, you actually change the policy to take it more often.

Notice what this means. Prediction produces a score. Control produces a decision. The score is useful precisely because it feeds the decision — but the score alone never makes the decision for you.

Here is the relationship that keeps the field coherent: control is the goal, and prediction is a tool that serves it. You evaluate policies because evaluation tells you where improvement is possible. You never evaluate purely for the joy of measuring.

The Boundary in One Table

When the distinction blurs, return to this table. It compresses the whole boundary into four questions.

If you are unsure which problem you are solving, ask the diagnostic question: Am I measuring a policy I will keep, or am I trying to change it? Measuring means prediction. Changing means control.

The Bridge: How a State Value Becomes an Action Choice

Before we walk through improvement, we need to close one gap. If you evaluate a policy and get V(s), you have a score for starting in state s and then following the policy. That score does not directly tell you which action to take first. A state value describes the policy's behavior from that point forward — it does not rank the actions available right now.

To compare actions, you need one of two things.

The first option is an action value, Q(s, a). This scores taking action a in state s, then following the policy afterward. Unlike V(s), which bundles the first decision into the policy, Q(s, a) holds that first decision fixed and lets you compare actions directly. If Q(s, left) = 5.0 and Q(s, right) = 8.0, the comparison is explicit: under the current policy's behavior after this moment, moving right looks better.

The second option is a one-step lookahead. If you do not have Q values, you can build the comparison yourself: for each candidate action, take the immediate reward you expect, add the discounted value of the state you land in, and compare those sums. In symbols, you are comparing r + γV(s′) for each action. This works because V(s′) already encodes everything the policy does after the next state — you only need to evaluate the one decision in front of you.

Here is the key insight: the policy's own value function supplies the continuation, and your comparison of first actions supplies the improvement. That is how evaluation becomes action selection. The value estimate does not magically contain an action ranking. It provides the future consequences; you compare the immediate choices against those consequences.

How Evaluation Becomes Improvement: The Loop

The boundary between prediction and control is real, but the two problems do not sit in isolation. They form a loop — and almost every reinforcement learning algorithm you will meet is built from that loop.

Here is the mechanism. First, evaluate the current policy to get value estimates. Second, use those estimates to improve the policy: compare the candidate actions in each state, and switch the policy to the action with the higher value. Third, evaluate the new policy. Its values will differ from the old one, because the agent now behaves differently. Fourth, improve again.

Why does this repeat? Because improving the policy changes the values, and changed values reveal new opportunities for improvement.

Walk through a tiny example. Imagine a two-state corridor. In state A, the agent can move left or right. The current policy says "move left." Evaluation under this policy gives you V(A) = 5.0 — the expected return from state A when the agent always moves left.

Now compare the first action. To evaluate moving right, you do not use V(A). You need the action value Q(A, right): the immediate reward for moving right, plus the discounted value of wherever that move lands you, assuming the policy takes over afterward. Suppose that calculation gives Q(A, right) = 8.0. The comparison is now concrete: following the policy from A yields 5.0, but taking the right action first and then following the policy yields 8.0. Improvement flips the policy: now the agent moves right.

But the new policy changes what happens after state A, which changes the expected return of moving right. Evaluate again. The value may rise to 9.0, or it may reveal that an even better option exists further down. The loop continues until the values and the policy agree with each other.

This cycle — evaluate, improve, repeat — is the engine underneath many algorithms you will encounter. The formal name is generalized policy iteration, but you do not need the formalism to grasp the shape: prediction and control are two phases of one repeating cycle. Evaluation tells you where you stand. Improvement moves you forward. Neither alone is sufficient.

One qualification matters. In a small environment where you can evaluate exactly, this loop can converge to a policy that is genuinely optimal — no alternative policy does better. In practice, with approximate or noisy estimates, the policy may stop changing because your estimates are imperfect, not because you have found the true best policy. The loop is a search, and the quality of the search depends on the quality of the measurements feeding it.

Common Mistakes Beginners Make at This Boundary

The prediction/control boundary produces a predictable set of errors. Knowing them in advance saves you debugging time later.

Mistake 1: Treating a single value estimate as proof the policy is optimal. A high value does not mean "best." It means "good under this policy, in this environment, with these rewards." The optimal policy is the one that maximizes expected return — and you cannot certify optimality from one score. You certify it by showing no alternative policy does better.

Mistake 2: Evaluating with the wrong policy in mind. Values only describe the policy that produced them. If you estimate values under a random exploration policy and then act greedily with respect to those values, you are mixing two different policies. The values describe the old behavior, not the new one. This is not a philosophical nitpick — it is the reason many algorithms require careful handling of which policy generated the data.

Mistake 3: Expecting evaluation alone to change behavior. This is the original confusion. Computing a value estimate changes nothing about what the agent does. The agent keeps following its old policy until something explicitly updates the policy. If your agent is not improving, check whether you ever wrote the improvement step — not just the evaluation step.

Mistake 4: Using V(s) to rank actions directly. A state value describes the policy's behavior from that state onward. It does not tell you which first action is best. To compare actions, you need Q(s, a) or a one-step lookahead: immediate reward plus the discounted value of the next state. Skipping this step is how beginners convince themselves that evaluation alone produces decisions.

Mistake 5: Confusing "the best action in a state" with "the best policy overall." Choosing the highest-valued action in one state is a local decision. The optimal policy is a global object: a complete decision rule for every state the agent might encounter. Local greediness can produce a globally bad policy if the values themselves are inaccurate.

When to Think in Prediction vs When to Think in Control

You will face both problems constantly — in your own experiments and in the papers and documentation you read. A simple decision rule keeps you oriented: ask whether you are measuring a policy you will keep, or trying to change it.

Use the prediction mindset when you are debugging. You wrote a policy, you want to know if it behaves sensibly, and you need a value estimate to check your expectations. Use it when you are comparing two fixed strategies — which baseline performs better on this environment? Use it when you are trying to understand an environment's reward structure before committing to a training run.

Use the control mindset when you are training. If your goal is an agent that acts well, you are solving a control problem. Tuning behavior, choosing among candidate policies, and improving performance all belong to the control side.

One more thing to normalize: real algorithms blur this line constantly. Many practical methods evaluate and improve simultaneously, updating value estimates and policy decisions in the same training step. That blurring is not a mistake — it is an engineering choice that makes learning efficient. But you will understand those algorithms far better if you can see which parts are doing evaluation and which parts are doing improvement.

Your Next Experiment: Watch Evaluation Become a Decision

Here is a concrete next step. Take a tiny environment — a small grid, a short corridor, anything you can simulate quickly. A small known environment works best, because you can calculate exact values instead of guessing from noisy samples.

First, fix a policy and evaluate it. Write down V(s) for each state. Second, pick one state and compare its candidate actions. For each action, calculate the immediate reward plus the discounted value of the next state under your fixed policy. Label those quantities Q(s, a). Third, if a different action scores higher than the policy's current choice, change the policy in that state. Fourth, evaluate the new policy. Watch the values shift.

That observable shift is the prediction/control boundary made visible — measurement turning into decision, and decision turning into a new measurement. If the values do not change, you have learned something too: your improvement step did not actually change the agent's behavior, which means the policy was already greedy with respect to its own values.

Name the question before you run the code. Are you asking how good is this policy? That is prediction. Or are you asking which policy should I use? That is control. The distinction sounds small. It organizes everything that follows.