Calculate RL Bootstrap Targets for Terminals and Time Limits

A training loop can run clean, print a falling loss, and still learn the wrong thing. The bug is often one boolean.

Here is the drill: four transitions, one small Python script, two different masks, and a printed gap that shows exactly where the error lives. By the end you will be able to point at a number and say which mask produced it.

This article assumes you already know what a reward, a return, and a value estimate are. We are not re-deriving those. We are looking at one line of arithmetic at the boundary of an episode, and at what happens when that line is wrong.

Why One "Done" Flag Breaks the Target

An episode can stop for two very different reasons.

The first: the modeled process reached an end state. The agent fell into the pit, crossed the finish line, or the game was lost. The task itself says there is no future here.

The second: something outside the task cut the episode off. A simulator's maximum step count fired. A wall-clock timer expired. The task would have continued, but an external limit said stop.

In a naive training loop, both events look identical. A single done boolean flips to True, the loop resets, and the next episode begins. Same signal, same code path, two different meanings.

That matters because of one term in the learning target: the bootstrap. The bootstrap is the part of the target that borrows the next state's value estimate. It is a bet that the future still exists. That bet should survive a time limit. It should not survive a true terminal.

Gymnasium's step API split the old done into terminated and truncated for exactly this reason. The old single flag hid the distinction, and a lot of implementations treated the two cases as identical. They are not.

Name the two failure directions now, because you will see both in the output:

• Treating truncation as termination undervalues every state near the arbitrary cutoff. The agent is told its future is worth nothing, right before the future would have continued.
• Treating termination as continuing invents future reward that the task definition says is zero. The agent learns to expect a tomorrow that does not exist.

One line of arithmetic decides which mistake you make.

The Target Formula, Written So You Can Code It

The whole thing fits in one expression:

target = reward + (bootstrap_allowed) * discount * next_state_value

The mask, bootstrap_allowed, is a boolean. It is not a small correction or a tuning knob. It is the presence or absence of an entire future.

Work a hand example with round numbers. Reward is 1, discount is 0.9, and the next-state value estimate is 5.

• Bootstrapping allowed: 1 + 1 * 0.9 * 5 = 5.5
• Bootstrapping blocked: 1 + 0 * 0.9 * 5 = 1.0

That is the full range of the decision. One number or the other, and nothing in between.

Two assumptions keep this exercise honest. First, the next-state value estimate is supplied as a fixed number, not computed by a network. The arithmetic is fully deterministic and you can check it on paper. Second, the mask follows the environment definition, not the presence of a timer. A horizon that is part of the task itself — a game that ends after ten moves — is termination. An external training limit imposed on an otherwise continuing task is truncation. Do not decide by checking whether a clock was involved.

Set Up the Exercise

Dependencies: Python 3 and nothing else. No RL library, no environment, no GPU, no credentials. The point is to isolate the target arithmetic from all the training machinery around it, so the only thing that can be wrong is your mask.

The input is a small hand-written list of transitions. Each one carries a reward, a next-state value estimate, a discount, and separate terminated and truncated booleans.

We need four rows to cover the interesting cases:

Row	Situation	terminated	truncated
A	Continuing transition	False	False
B	True terminal	True	False
C	External time limit	False	True
D	Both flags set	True	True

Row D is the one people forget. When both flags fire, which one wins? Hold that question; the code will answer it.

Success criteria: the script prints a table of targets, and you can point at each number and say which mask produced it. If you can do that for all four rows, the exercise worked.

Run It: Correct Masks First

Here is the smallest useful implementation.

transitions = [
    # label, reward, discount, next_value, terminated, truncated
    ("A continuing",  1.0, 0.9, 5.0, False, False),
    ("B terminal",    1.0, 0.9, 5.0, True,  False),
    ("C truncated",   1.0, 0.9, 5.0, False, True),
    ("D both",        1.0, 0.9, 5.0, True,  True),
]

def target(t, bootstrap_allowed):
    _, reward, discount, next_value, _, _ = t
    return reward + bootstrap_allowed * discount * next_value

print(f"{'row':<14}{'term':<7}{'trunc':<7}{'correct':<10}{'wrong':<10}")
for t in transitions:
    label, reward, discount, next_value, terminated, truncated = t

    # Correct mask: bootstrap unless the process truly ended.
    correct = target(t, not terminated)

    # Wrong mask: collapse both flags into one end-of-episode signal.
    done = terminated or truncated
    wrong = target(t, not done)

    print(f"{label:<14}{str(terminated):<7}{str(truncated):<7}"
          f"{correct:<10.2f}{wrong:<10.2f}")

Expected output:

row           term   trunc  correct   wrong
A continuing  False  False  5.50      5.50
B terminal    True   False  1.00      1.00
C truncated   False  True   5.50      1.00
D both        True   True   1.00      1.00

Read the columns. The correct mask bootstraps on rows A and C, and blocks it on rows B and D. The wrong mask bootstraps only on row A.

Notice what the code is doing. The mask is a statement about the task — does the future exist after this transition? — and the expression not terminated makes that statement visible in one place. When someone later asks why row C keeps its bootstrap term, you point at that line.

Row D deserves a pause. The mask not terminated blocks the bootstrap there, which is the right answer if terminated = True correctly records that the task genuinely ended. But a record with both flags set is a conflicting signal, not a clean one. It usually means the environment set the flags inconsistently, or that a wrapper added a time limit on top of an already-finished episode. Before you trust the mask on a row like this, check the environment definition and confirm which flag is telling the truth. The mask is only as good as the flags feeding it.

Verify Two Values by Hand

Do not trust the printout yet. Compute two of these on paper.

Row A, continuing. Reward 1, discount 0.9, next-state estimate 5. The future exists, so the bootstrap is present: 1 + 0.9 * 5 = 5.5. The script printed 5.50. Match.

Row B, true terminal. Same reward, same discount, same next-state estimate — but the process ended. The bootstrap is gone: 1 + 0 = 1.0. The script printed 1.00. Match.

Now do row C yourself. Reward 1, discount 0.9, next-state estimate 5, and truncated = True. If you wrote 5.5, you understood the mask. The truncated row's target is numerically identical to the continuing row, and that is correct, not a bug. The time limit interrupted the episode; it did not end the task. The future still exists, so the target still borrows from it.

This is the step where pattern-matching the code stops working and the mental model has to carry the weight. If row C surprised you, go back and re-read the mask expression before continuing.

Now Break It: The Wrong Mask

The script already printed the wrong column. Look at it again:

row           term   trunc  correct   wrong
A continuing  False  False  5.50      5.50
B terminal    True   False  1.00      1.00
C truncated   False  True   5.50      1.00
D both        True   True   1.00      1.00

Rows A, B, and D are unchanged. Only row C moved, and it moved down by exactly discount * next_value = 0.9 * 5 = 4.5.

That gap is the error, and it is not small. Every state near the arbitrary cutoff is being told its future is worth nothing. The agent learns to rush toward the time limit, or to fear it, depending on the reward structure — either way, it learns something the task never said.

Common mistake: Collapsing terminated and truncated into a single done boolean. It looks harmless because the loop resets either way. It is not harmless, because the target does not.

There is a second failure mode worth seeing, and it hides even when the mask is right. Many environments auto-reset: when an episode ends, the next step() call returns the first observation of the new episode. If you store that observation as the "next state" in your replay record, you have silently corrupted the bootstrap. The mask says "bootstrap allowed," but the value estimate it borrows from belongs to a different episode entirely.

The fix is to keep the final observation of the transition, not the first observation of the next episode. This is easy to lose and hard to notice, because the numbers still look plausible.

When the Mask Should Not Be a Single Boolean

You now have a rule. Do not over-apply it.

Use the terminated-only mask when the environment definition says the process genuinely ended. Do not use it merely because a step counter fired.

Use the truncated-aware mask when an external limit interrupted a task that would otherwise continue.

When the horizon is part of the task itself — a fixed number of moves, a controller with a set action budget — remaining time belongs in the state, and the end of the horizon is termination. The flag follows the definition, not the timer.

Continuing tasks with no end never set either flag, and the bootstrap term is always present.

The practical rule: write your bootstrapping policy next to the environment definition, in the same file or the same comment block. A later refactor that changes how episodes end should not be able to quietly change your learning objective. If the policy is written down where the environment lives, the refactor has to argue with it.

Your Turn: Two Modifications

First, change the discount factor to 0.99 and predict which rows move before you run anything. The terminal rows should not move at all — their target is just the reward. The continuing and truncated rows should move, and they should move together.

Second, keep the discount at 0.9 but raise the next-state value estimate to 50 on every row. Before running, write down the new gap between the correct and wrong columns for row C. The gap is discount * next_value, so it should scale from 4.5 to 45. Run it and check. This is the point of the exercise: the size of the truncation error is not fixed. It grows with the discount and with how much future value the state actually has, which is why the mistake gets more expensive in long-horizon tasks.

Optional extension: wrap the arithmetic in a function that takes a transition record and returns the target. Keep the mask logic in that one function so it lives in a single testable place. When the target calculation is one function, a wrong mask is one line to fix instead of a search through the training loop.

The mask is a claim about the task. The printed gap between the two columns is the price of getting that claim wrong. Keep terminated and truncated separate in your replay records, store the final observation of each transition, and write the bootstrapping policy next to the environment definition — so the arithmetic you just verified survives contact with a real training loop.