Rewards, Returns, and Discounting: What an RL Agent Is Actually Optimizing

A reward is feedback. The return is the scoreboard. Beginners often treat them as the same thing—and then wonder why their agent grabs the pebble instead of crossing the maze for the gold.

Imagine a robot in a simple maze. Ten steps ahead sits a large gold coin worth +100. Right next to the robot's foot is a small pebble worth +1. If the robot only cares about the reward in front of it, it grabs the pebble and stops. That robot looks busy, but it is not learning anything useful.

This is the trap at the heart of reinforcement learning. Newcomers often assume an RL agent is trying to maximize each individual reward it receives. In reality, the agent is trying to maximize something bigger: the total accumulated reward over time, called the return. Understanding the difference between a single reward and the return is the difference between an agent that grabs pebbles and one that crosses the maze for the gold.

The Reward Is Not the Goal

A reward is a single scalar number the environment sends back after each action. It is momentary feedback: "That move was good" or "That move cost you." The reward function in reinforcement learning defines how the environment scores each step, but it does not define what the agent should do next.

Here is the subtle part. If an agent tried to maximize every immediate reward in isolation, it would make terrible long-term decisions. Consider a delivery drone navigating a city. A reward function that gives +1 for moving toward the nearest package drop-off point sounds sensible. But if the nearest drop-off is trivial while a farther one pays ten times more, the drone that chases immediate reward will happily service the easy stop forever and never attempt the valuable route.

The reward is feedback. The return is the scoreboard. An agent that optimizes each reward independently is like a chess player who captures the first available pawn every turn without ever thinking about checkmate.

From Single Rewards to the Return

The return is the total accumulated reward an agent collects from a given point onward. If an agent takes four steps and receives rewards of +1, +2, -1, and +5, the return from the first step is simply:

1 + 2 + (-1) + 5 = 7

That sum is what the agent cares about—but only when the task has a clear ending. A single reward tells the agent how the environment felt about one moment. The return tells the agent whether the whole journey was worthwhile.

Reinforcement learning tasks come in two flavors, and each flavor changes how we define the return.

Episodic tasks have a natural ending point. A game of chess ends in checkmate. A maze run ends when the robot escapes. A CartPole episode ends when the pole falls. For these tasks, the return is the sum of rewards from the current step until the episode terminates. Simple and clean.

Continuing tasks have no natural end. A thermostat controls temperature indefinitely. A stock-trading agent operates as long as the market is open. There is no final step, which creates a problem: if the agent receives +1 every step forever, the sum of rewards grows without bound. An infinite return is not a useful optimization target.

This is where discounting enters the picture.

Why Discount Future Rewards

The discount factor, typically written as gamma (γ), is a number between 0 and 1 that shrinks the value of future rewards. A reward received one step from now is worth γ times its face value. A reward two steps away is worth γ² times its face value. Three steps away, γ³. The further into the future a reward sits, the less it contributes to today's return.

The discounted return formula looks like this:

G = r₁ + γr₂ + γ²r₃ + γ³r₄ + ...

Each future reward gets multiplied by γ raised to the power of how many steps away it is.

Why do this? Two reasons, one mathematical and one practical.

Mathematically, discounting keeps infinite sums finite. If γ = 0.9 and the agent receives +1 every step forever, the return converges to 1 / (1 - 0.9) = 10 instead of exploding to infinity. The geometric series settles because each future term contributes less than the one before it.

Practically, discounting is a modeling choice: it lets you decide how much delay matters. A dollar today is worth more than a dollar next year because you can use it now, invest it, or avoid the risk that the future never arrives. The same logic applies to an RL agent. A reward ten steps away is less certain and less useful than an identical reward arriving now.

The discount factor also controls how far-sighted the agent becomes. A low gamma, like 0.5, makes the agent heavily myopic: rewards more than a few steps away barely register. A high gamma, like 0.99, makes the agent patient: it will tolerate long delays for a large eventual payoff.

Note: Gamma is not a statement about what is objectively valuable. It is a dial you set to express how much you want the agent to care about delayed consequences. Two designers can set different gammas for the same task and both be right—they just want different behavior.

What Gamma Actually Changes in an Agent's Priorities

Let me show you why gamma is not a minor technical detail. It changes which behavior the agent prefers.

Imagine an agent facing two paths. Path A delivers +10 immediately, then nothing. Path B delivers nothing for five steps, then +20.

With γ = 0.5, the return for Path A is 10. The return for Path B is 0 + 0 + 0 + 0 + 0 + (0.5⁵ × 20) = 0.625. The agent picks Path A without hesitation. The distant reward has been discounted into near irrelevance.

With γ = 0.95, Path A still returns 10. Path B now returns 0.95⁵ × 20 ≈ 15.5. The agent picks Path B. The same scenario, the same rewards, but a different gamma produces a different decision.

This is why gamma appears as a hyperparameter in nearly every RL algorithm you will meet. When you see GAMMA = 0.99 in code, you are looking at a statement about how patient the agent should be.

Even in episodic tasks with a clear ending, practitioners usually still discount. The reason is practical: discounting changes the objective so that delayed completion is no longer equivalent to fast completion. A maze-solving agent with no discount has no incentive to finish quickly, only to finish eventually. Add a small discount, and the agent suddenly cares about the number of steps it takes, because each extra step shrinks the value of the final reward.

Common mistake: Discounting does not automatically make an agent faster. It makes the agent prefer sooner rewards over later ones. Whether that produces faster goal-reaching depends on your reward structure. If you want speed, make sure the return actually rewards it.

When Reward Design Backfires

Here is where the reward-versus-return distinction becomes a practical debugging skill rather than an abstract concept.

An RL agent does not optimize what you intended. It optimizes whatever return your reward function implies. Those two things can diverge spectacularly.

The classic failure pattern is rewarding a proxy instead of the real goal. Suppose you want a robot to reach a target location. You give +1 for every step that moves it closer to the target. This seems helpful, but watch what happens: the agent can earn reward indefinitely by cycling between two positions, moving closer and then backing away, never actually arriving. The proxy reward (closer is better) has replaced the real goal (arrive at the target).

I have seen this exact failure in practice. A grid-world agent was given +1 for moving closer to a target and -1 for moving away, with +10 for actually acquiring the target. The agent learned to walk into a wall. Why? Because moving toward the wall counted as "closer" to the target in the reward function's geometry, and the agent discovered it could farm small positive rewards by shuffling against the boundary. The reward function was technically being maximized. The intended behavior was nowhere in sight.

Sparse rewards create the opposite failure mode. If you reward only at the goal and give zero everywhere else, the agent receives no feedback to guide it toward the goal. It wanders randomly, hoping to stumble onto the reward. Dense rewards guide learning but invite gaming. Sparse rewards resist gaming but make learning painfully slow.

The diagnostic habit that saves you: before running a training loop, ask what behavior maximizes the return under this reward function. Trace the best and worst plausible behaviors by hand. If the best behavior is not what you actually want, the reward function is wrong—and no amount of training will fix it.

Note: This diagnosis applies to objective mismatch. If your agent performs poorly despite a reward function that seems to capture what you want, the problem may be exploration, representation, or optimization instead. But when a behavior scores best under your stated return while violating your intent, the reward design is the culprit.

A Simple Way to Check Your Own Reward Design

You do not need a full training run to test a reward function. You need a pencil and a willingness to trace a few trajectories.

Pick a small task. Write out a short imagined sequence of states and actions. Assign rewards to each step. Compute the discounted return by hand under two different gamma values. Then ask the uncomfortable question: does the behavior that maximizes this return match the behavior you want?

Try this with a toy problem of your own. A one-dimensional walk where the agent moves left or right, with a reward at the far end, is enough. Write out the rewards for the direct path and for a stalling path. Compute both returns. Watch how gamma flips which path wins.

This calculation is not a ritual. It is a debugging tool. It forces you to see the return your reward function actually implies, rather than the one you hoped it implied. The skill transfers directly to everything that comes next: value functions, Q-learning, and policy gradients all build on this same foundation of accumulated, discounted reward.

The agent is not optimizing the reward in front of it. It is optimizing the return that stretches ahead of it. Learn to compute that return, understand what gamma does to it, and you will be able to predict what your agent will do before it ever takes a step.