Value Functions in Reinforcement Learning: Predicting Future Reward

A reward tells you what just happened. A value function predicts the return you can expect from here to the end of the episode—and that forward-looking prediction is what makes deciding possible.

Why a Reward Is Not Enough to Decide

Here is the trap every reinforcement learning beginner walks into right after learning about rewards: if the agent just maximizes reward, won't it figure everything out?

Not quite. Think about the timing problem. A reward arrives after an action. It is feedback about what already happened. But the agent has to choose before it knows the outcome. Standing at a fork in the road, the agent cannot feel the reward for the path it has not taken yet. It needs a guess about the future.

That guess is the job of a value function.

If you have already met returns and discounting, you know that rewards accumulate into a single number called the return—the total discounted reward the agent expects to collect over time. And if you have met policies, you know they are the rules that turn states into actions. Value functions sit between those two ideas: they answer the question, if I follow this policy from here, what return should I expect over the rest of the episode?

The agent's real question is never "was that good?" It is always "if I stand here, what comes next?"

The State Value Function V(s): How Good Is This Situation?

The state value function, written as V(s) or V_π(s), answers exactly that question. It predicts the expected return the agent will collect if it starts in state s and follows a particular policy from that point forward.

Notice the phrase "particular policy." This is the detail that trips up most beginners, so let me slow down here.

V is not a property of the state alone. A state does not carry a fixed price tag. The same state can be valuable under one policy and worthless under another, because the value depends on what the agent does next.

Imagine a two-step path. From state A, the agent can move to state B, which always pays +10. From state B, it can move to state C, which always pays −50. A policy that stops at B collects +10. A policy that keeps walking to C collects +10 − 50 = −40. State B is not "worth" anything on its own. It is worth +10 under the first policy and −40 under the second.

That is why you will see V written with a subscript: V_π(s). The subscript is not decoration. It is a reminder that the value function is married to the policy. Change the policy, and the values change with it.

This also explains why a state can look attractive now and still be a trap. A chess square that captures a pawn looks great—until you notice it opens your king to attack. The immediate reward is positive, but the expected return is negative because of what comes after. V captures that whole future, not just the next step.

The Action Value Function Q(s, a): How Good Is This Move?

The action value function, written as Q(s, a) or Q_π(s, a), is the finer-grained cousin of V. It answers a slightly different question: if I take action a in state s, and then follow the policy afterward, what return should I expect?

Here is the exact difference between V and Q. V averages over whatever actions the policy might pick. If the policy is 70% likely to move left and 30% likely to move right, V blends both possibilities together. Q does not blend. Q pins down one specific action first, then asks what the future looks like after committing to it.

Think of it this way. V asks: how good is this situation, given how I usually behave? Q asks: how good is this specific move, before I fall back on my usual behavior?

That distinction makes Q the more useful tool for comparing actions. Suppose you are in a state with two possible actions. Action 1 leads to a small reward now and a dead end. Action 2 leads to no reward now but opens a corridor of large rewards. V alone cannot tell you which action to take, because it has already averaged over both. Q keeps the two options separate, so you can compare them directly.

Note: Knowing Q values lets the agent rank actions, but ranking is not the same as choosing. A separate policy decides how to use that ranking. Greedy selection—always taking the highest-Q action—is one option, and during learning it is usually mixed with exploration so the agent keeps testing alternatives.

V and Q: Same Idea, Different Question

Here is the relationship that makes everything click: V and Q are not two unrelated ideas. They are two zoom levels of the same prediction.

Under a given policy, V(s) is the policy-weighted average of Q(s, a) over all the actions the policy might choose. If the policy always picks action 1, then V(s) equals Q(s, action 1). If the policy is stochastic, V(s) blends the Q values according to the policy's probabilities.

Q(s, a), in turn, is the value of committing to one action before returning to policy-following. It is V measured at a finer resolution.

When should you reach for each? If the agent needs to compare what to do right now, Q is the natural tool, because it separates the options. If you want to evaluate whether a state is promising—or whether one policy performs better than another—V is often the cleaner measure.

Common mistake: Treating V and Q as two separate families of ideas. They are not. They are the same prediction at different resolutions: V looks at the situation, Q looks at the move.

How a Value Estimate Gets Built: The One-Step Look

You now know what V and Q predict. The next question is how those predictions get refined. The core idea is almost suspiciously simple: the value of where you are now equals the reward you expect immediately, plus the discounted value of wherever you land next.

This is the Bellman idea, and you can see it with one tiny calculation.

Suppose your discount factor γ is 0.9. From state A, you take an action that pays +10 and leads to state B. You already believe V(B) = 50. Then your updated estimate for the value of being in A is:

V(A) = 10 + 0.9 × 50 = 55

Why discount the 50? Because a reward collected later is worth less than one collected now. The discount factor shrinks future value accordingly—that is the same discounting you met with returns.

Now imagine you receive new experience that changes your belief about B. If V(B) drops to 30, then V(A) drops too:

V(A) = 10 + 0.9 × 30 = 37

That is the learning loop in miniature. The agent does not start with accurate values. It starts with guesses, takes actions, observes rewards, and revises its estimates. Every piece of experience is evidence that updates V and Q. Over time, the guesses sharpen into predictions the agent can trust.

Why Value Estimates Beat Immediate Reward

Now we get to the payoff. Why does any of this matter?

Because an agent that only chases immediate reward gets trapped. It grabs the shiny coin in front of it and walks into the dead end. It takes the easy win and sacrifices the corridor of larger rewards behind it. A reward-maximizing agent that looks only one step ahead is, in practice, a short-sighted agent.

Value functions fix that by letting the agent compare whole futures instead of single rewards.

Picture a simple scenario. Two doors. Behind door 1: a reward of +5 right now, then nothing. Behind door 2: a reward of 0 right now, then +20, then +20 more. The immediate-reward agent picks door 1 every time. The value-aware agent looks at the expected returns—+5 versus +40—and picks door 2 without hesitation.

That is the entire point of value functions. They convert "what will happen later" into a number the agent can compare now. They make the future legible at the moment of decision.

If you continue into reinforcement learning, you will find that a whole family of algorithms—Q-learning, SARSA, deep Q-networks—are really just different machinery for computing and refining these same estimates. The one-step look you just worked through is the seed they all grow from.

Common Misconceptions and How to Avoid Them

Let me close with four common mental-model errors worth catching early.

Misconception 1: V(s) is a fixed property of the state. If you catch yourself thinking a state is simply "good" or "bad" on its own, check your model. Value depends on the policy. The same state can be a goldmine under one policy and a trap under another. The symptom that reveals this error: you are surprised when changing the policy changes the values.

Misconception 2: Value equals the next reward. Value is the whole discounted future, not the immediate payoff. If you find yourself computing V as if it were just the reward at the next step, you have collapsed the future into the present. The symptom: you cannot explain why a state with zero immediate reward might still have high value.

Misconception 3: Higher immediate reward always means the better action. This is the trap from the two-door example. The action with the bigger immediate reward can open a worse future. The symptom: you keep choosing short-term wins and wondering why the total return stays low.

Misconception 4: V and Q are interchangeable. They are related, but they answer different questions. If you need to compare actions, Q is the tool. If you need to judge a situation or evaluate a policy, V is often cleaner. The symptom: you know a state is good but cannot figure out which action to take from it.

Your Next Step

Here is the exercise I would do next, and I think it will cement everything faster than any amount of reading.

Take a small environment you already understand—a tiny grid, a board game, even a simple video game level. Pick a fixed policy, even a bad one. Write down V(s) by hand for three or four states: if I start here and follow this policy, what return do I expect? Then write down Q(s, a) for two different actions from the same state: if I commit to this action, then follow the policy, what do I expect?

Then change the policy. Make it greedier, or more random. Watch how the values change with it.

That observation—values shifting as the policy shifts—is the bridge to Bellman reasoning. Once you see value as a prediction that depends on policy and propagates through future states, you are ready for the equations that make the prediction precise.