Deriving RL Targets for Terminal States and Time-Limit Truncations

Your training loop ends every episode the same way. One flag flips, you reset, you move on. Somewhere in that single flag, two completely different events got merged — and one of them is quietly teaching your agent a lie.

Here is the lie in one line: target = reward + gamma * V(next_state). That bootstrap term, gamma * V(next_state), is the only part of the target that changes depending on why the episode ended. Get the boundary type wrong and you either erase future reward that still exists or invent future reward that never will. This article derives the correct target for each boundary so you can stop guessing.

I'm assuming you already know the agent-environment loop, what episodes and terminal transitions are, and what V and Q predict. If those feel shaky, the two prerequisite pieces on episodes and value functions are worth a pass first. Everything below builds on them.

The One-Line Target That Hides a Decision

Write the general one-step target:

$y = r + \gamma V(s')$

Read each symbol as a piece of the mechanism:

• $r$ is the reward received on this transition.
• $\gamma$ is the discount factor, in $[0, 1]$.
• $V(s')$ is your current estimate of the value of the next state.
• $\gamma V(s')$ is the bootstrap term — the part that carries the future into the present.

The reward $r$ is fixed by the environment. The discount $\gamma$ is a design choice. The bootstrap term is where the boundary decision lives. Whether $s'$ can still produce future reward determines whether that term survives or gets zeroed.

Note: The environment's MDP definition decides whether a boundary is terminal. A training-time limit does not change the task's MDP — it only changes when you stop observing it.

There are exactly three cases to derive: a continuing transition, a true terminal transition, and an externally truncated transition. They differ in one place — the bootstrap term.

Termination, Truncation, and the MDP Boundary

Before deriving, pin down what each boundary actually is, because the names in your code will mislead you.

Termination means the modeled process reached an absorbing end state. The task is over by definition — success, failure, or a genuine absorbing state. Future return after it is zero because the task says so.

Truncation means observation stopped for a reason outside the modeled process. The usual cause is a maximum step count. The underlying state is still valid; the environment could continue from it if you let it. Nothing about the task ended — your patience did.

Modern step APIs separate these into two flags — terminated and truncated — precisely because bootstrapping depends on the difference. Collapsing them into one done flag throws away the information the update needs.

Common mistake: A timer in the code does not automatically mean truncation. If the horizon is part of the task — a game that ends after ten moves, a controller with a fixed action budget — then the end is genuine, and remaining time may belong in the state. A training limit imposed on an otherwise continuing task is truncation. Read the environment definition, not the variable name.

That is the decision rule for this whole article: classify the boundary from the environment definition, not from what your loop happens to call it.

Deriving the Target for Each Boundary

Now derive each case from the return definition rather than asserting a rule.

Continuing transition. The process keeps going. The return from $s'$ is exactly what $V(s')$ estimates, discounted one step:

$y = r + \gamma V(s')$

The bootstrap stays.

True terminal transition. By definition, the return after a terminal state is zero — there is no future. So the discounted future term is $\gamma \cdot 0 = 0$, and:

$y = r$

The bootstrap must be zeroed. This is not a convention; it falls out of the return definition.

Truncated transition. The process did not end. The state is valid and could continue, so the best estimate of future return is still $V(s')$:

$y = r + \gamma V(s')$

The bootstrap stays — identical to the continuing case.

Notice the truncated case ends the training episode (you reset) even though it does not end the return. Those are two different things, and conflating them is the source of the bug.

To express all three in one update, introduce a bootstrap mask $m \in \{0, 1\}$:

$y = r + \gamma \, m \, V(s')$

Set $m = 0$ for true terminals, $m = 1$ for continuing and truncated transitions. The mask is not a new idea — it is bookkeeping for the derivation you just did.

Worked Numbers: Same Reward, Three Different Targets

Fix one transition: $r = 1$, $\gamma = 0.9$, $V(s') = 5$.

Continuing and truncated agree at 5.5. The terminal target is 1.

Now watch the two mistakes:

• Treating a truncated transition as terminal gives $1$ instead of $5.5$. You told the agent there is no future from a state that still has one. That is a systematic undervaluation of states near the arbitrary cutoff.
• Treating a terminal transition as continuing gives $5.5$ instead of $1$. You invented future reward the task says cannot exist — over-optimistic values that keep the agent chasing ghosts.

The truncation error is a bias, not noise. It pushes estimates in one direction. The direct damage happens at the truncated transition itself: the target loses the estimated future from that state. Whether that error then spreads backward to earlier states depends on your learning updates and which transitions you actually sample — it is a likely consequence, not an automatic one.

Where the Error Actually Bites

Small, fast-converging tabular problems often absorb the truncation error without visible damage. That is exactly why the bug survives code review — the demo works.

It bites harder when:

• The time limit is used to explore regions the agent has not reached yet.
• The cutoff is short relative to the task, so many episodes end by truncation.
• The discount factor is high, so the erased bootstrap term was large.

The symptom pattern is recognizable once you know to look: value estimates that flatten or sag near the step limit, and a policy that avoids states close to the boundary. Treat those as possible symptoms, not guaranteed ones — the exact pattern depends on how the error propagates through your updates. The reverse error on true terminals produces over-optimistic values and reward-chasing behavior.

Warning: The exact magnitude depends on the environment, the discount factor, and how often the cutoff is hit. Do not expect a fixed number — expect a direction.

Storing Boundaries Correctly in a Replay Buffer

The theory is easy to get right and easy to lose again in the data layer.

A replay record should preserve the reward, the next observation, and separate terminated and truncated flags — not one collapsed done. If you collapse them at storage time, no amount of correct math downstream can recover the distinction.

The auto-reset hazard is the one that gets people. When the environment resets automatically, the stored next observation can become the first observation of the next episode instead of the final observation of the transition. Your bootstrap then evaluates the wrong state entirely.

Write your bootstrapping policy next to the environment definition so a later refactor cannot silently change the learning objective.

A Compact Test You Can Run Before Training

Turn the derivation into a checkable habit. Construct one continuing, one terminal, and one truncated transition with identical reward and next-state estimate, then assert the expected targets explicitly:

r, gamma, v_next = 1.0, 0.9, 5.0

def target(r, gamma, v_next, terminated, truncated):
    mask = 0.0 if terminated else 1.0
    return r + gamma * mask * v_next

assert target(r, gamma, v_next, False, False) == 5.5  # continuing
assert target(r, gamma, v_next, False, True)  == 5.5  # truncated: keep bootstrap
assert target(r, gamma, v_next, True,  False) == 1.0  # terminal: zero it

Then test the auto-reset path specifically, because the final observation is the easiest thing to lose. This check is cheap and far more informative than another long training run when the target itself is suspect. A model cannot compensate for a target that erases the wrong future.

The Decision Rule

Classify the boundary from the environment definition. Keep the bootstrap for continuing and truncated transitions. Zero it for true terminals. Verify with the three-transition test before you trust a training curve.

That is the whole derivation compressed into a habit. The next step is to drop these targets into a full TD or Q-learning update and watch how the mask changes the update rule you write — same mechanism, one more symbol.