Episodes, Terminal States, and Horizons in Reinforcement Learning

You can follow the agent-environment loop: the agent sees a state, picks an action, gets a reward, and lands in a new state. Then it happens again. And again. But here's the question that trips up nearly every beginner: when does this loop actually stop?

The answer isn't always obvious. A chess match ends. A pole-balancing run ends when the pole falls. A thermostat never ends. If you treat all three the same way, your return calculations and your evaluation results will quietly go wrong. This article gives you the vocabulary to tell these situations apart.

The Agent Loop Has a Shape

In the previous article, you saw the basic rhythm of reinforcement learning: the agent observes a state, chooses an action, receives a reward, and transitions to the next state. That loop is the heartbeat of every RL problem.

But the loop itself doesn't tell you when to stop counting. Nothing in the cycle of state-action-reward-next-state says "this run is finished." That information comes from outside the loop, from the structure of the task you've defined.

To describe that structure, you need to answer two separate questions:

1. Does the task have a natural ending? If yes, each run is an episode that ends at a terminal state.
2. Is there a maximum interaction window? If yes, the task has a finite horizon that cuts runs short.

These are different dimensions, not competing categories. A chess game is episodic and finite-horizon if you cap it at 200 moves. A thermostat is continuing, but you might still evaluate it in fixed windows. Keeping the two questions separate is the key to getting the rest right.

What an Episode Is

An episode is one complete run of interaction, from a starting state to a finish. Think of it as one full attempt.

In chess, one episode is one game. It starts with the pieces in their initial positions and ends with checkmate, resignation, or a draw. In a video game, one episode is one round: you start at level one and play until your character dies or you beat the boss. In pole-balancing, one episode is one attempt to keep the pole upright, starting from vertical and ending when the pole falls past a critical angle.

Here's the distinction that matters: a time step is one action-and-reward exchange, while an episode is the whole sequence of steps from start to finish. A single chess move is a step. The entire game — all the moves strung together — is an episode. Episodes in reinforcement learning are built from many individual time steps.

After an episode ends, the environment resets, and a new episode begins from a starting state. This reset is what makes episodes useful: each one gives you a clean, self-contained sample of behavior.

Why does this matter for training? Because RL agents are usually evaluated by averaging results across many episodes, not by looking at single steps. One lucky move tells you nothing. The average score over a hundred complete games tells you whether the agent is actually learning.

Terminal States and Terminal Transitions

What makes an episode end? The agent reaches a terminal state — a state from which no further decisions are made.

The step that lands the agent in that terminal state is called a terminal transition. It's the final action-and-reward exchange of the episode. After that, the loop stops, and no further action is taken from the terminal state.

Here's the nuance beginners often miss: terminal states carry meaning, and that meaning varies. Winning a game of chess and losing a game of chess both end the episode. Both are terminal states. But they mean opposite things. One says the goal was reached. The other says the agent failed.

The pole-balancing example makes this concrete. The episode ends when the pole falls over too far or the cart runs off the track. That's a failure terminal state. But you could just as easily design a task where the terminal state is success — say, a robot arm that must place a block in a target position, where the episode ends the moment the block is placed.

A common source of confusion is the done flag you'll see in RL code libraries. When an environment returns done = True, it usually means the episode ended. But that flag alone doesn't tell you whether the agent succeeded or failed. You have to read the environment's rules to know which kind of ending you're looking at. Treating every done signal as success — or every one as failure — will distort your evaluation.

Finite Horizons: When the Clock Runs Out

Some episodes end not because the agent reached a natural terminal state, but because time ran out.

This is called a finite horizon: the task has a fixed maximum number of steps. A robot that must complete a task within 100 steps faces a finite horizon. A game round capped at 60 seconds faces one too. Even if the agent is doing perfectly well at step 99, the episode stops at step 100.

In modern RL libraries, this situation is called truncation. The episode was cut short for a reason other than reaching a terminal state. The environment might return a truncated = True flag to signal that the time limit was hit, distinct from terminated = True, which means the agent reached a natural ending.

Why does this distinction matter? Because a truncated episode is not a failure. If your robot was still making progress when the clock ran out, treating the truncation as a terminal failure will make your agent look worse than it actually is. Return calculations must handle the boundary correctly: the episode ended, but not because the task was completed or failed.

Evaluation code that mixes truncation with true termination can misjudge how well the agent is doing. If you're comparing two agents and one consistently gets cut off by the time limit just as it's about to succeed, you need to know that — otherwise you'll conclude it's worse than it really is.

Common mistake: A finite horizon is not a third kind of task sitting beside episodic and continuing. It's a boundary you can impose on either one. An episodic game can have a move limit. A continuing process can be sampled in fixed windows for evaluation. Ask "does it end naturally?" and "is there a time cap?" separately.

Continuing Tasks: When Interaction Never Ends

Some problems have no natural terminal state at all. The interaction could go on indefinitely.

A thermostat regulating room temperature is the classic example. There's no "end" to temperature regulation — the task is to keep responding to changes forever. Portfolio rebalancing works the same way: markets keep moving, and the agent keeps adjusting. Recommendation systems run continuously, serving suggestions as long as users keep engaging.

These are called continuing tasks, and they create a mathematical problem.

Remember how returns work: the return is the sum of rewards over time. If a task never ends, and you try to sum rewards without any stopping point, the total can grow without bound. An undiscounted sum over infinite steps is infinite — not a useful number for learning.

This is where the discount factor earns its keep. By multiplying future rewards by a discount factor (usually denoted γ, with a value between 0 and 1), distant rewards contribute less than immediate ones. The discounted sum stays finite, and the agent can meaningfully compare different courses of action.

The pole-balancing problem shows how flexible this framing can be. You can treat it as an episodic task: each attempt to balance the pole is one episode, and the return is the number of steps until failure. Or you can treat it as a continuing task: the pole never stops needing balance, and you use discounting to keep the return finite. Both framings are valid. Both can lead the agent to the same behavior — keeping the pole upright as long as possible. But the math you use to compute returns differs.

The framing is a modeling choice, not a fact about the world. And it has real consequences for your calculations.

Note: Discounting is one common way to handle continuing tasks, but it's not the only one. Some formulations use average reward per step instead. The important thing is to choose an explicit objective and evaluation criterion — not to assume discounting is the automatic answer.

Choosing the Right Boundary for Your Task

When you sit down with a new RL problem, separate the two questions:

1. Does the task have a natural end state? If yes, it's episodic. The episode ends when the agent reaches that state — whether it's success or failure.
2. Does the task impose a time cap? If yes, there's a finite horizon. The episode can end by truncation even if no terminal state was reached.

If the answer to both is no, the task is continuing. Interaction goes on indefinitely, and you need an explicit objective — often discounted return — to keep the numbers meaningful.

Notice that both questions can be true at once. A chess game with a move limit is episodic and finite-horizon. A continuing environment wrapped with a step limit for training is still continuing at its core; you've just added an administrative cutoff. The label you use depends on which boundary you're talking about.

Three mistakes tend to follow from picking the wrong frame:

Mistake 1: Treating a truncated episode as a true terminal state. This distorts evaluation. A time-limit cutoff is not a failure signal, and your return calculations shouldn't treat it like one.

Mistake 2: Summing rewards over a continuing task without an explicit objective. You'll get an unbounded return, which gives your agent no useful learning signal.

Mistake 3: Assuming every done signal means success. The done flag tells you the episode ended. It doesn't tell you why. Read the environment's rules to know whether you're looking at a win, a loss, or a time-out.

The boundary you choose determines how you compute returns and how you judge whether the agent improved. Get it wrong, and you'll draw confident conclusions from broken math.

A Quick Way to Keep the Boundaries Straight

Here's the compact version I keep in my head:

• A step is one action-and-reward exchange.
• An episode is a run that ends at a terminal state.
• A terminal transition is the step that lands in that state.
• Truncation is a cutoff from a time limit or other external rule.
• A finite horizon is a maximum interaction window.
• A continuing task has no natural ending.

One last reminder: the same environment can sometimes be framed multiple ways. Pole-balancing can be episodic or continuing depending on how you define the task. That's not a cheat — it's a modeling choice with real mathematical consequences. Choose deliberately.

Your Next Step

Before you run any RL experiment, look at the environment you plan to use and answer two questions:

1. Does it have a natural end state?
2. Does it impose a time limit?

Then check whether the return calculation and evaluation code you plan to use match those answers. If the environment is episodic, compare episode returns and lengths under the same termination and truncation rules. If it has a time limit, handle truncation separately from true termination. If it's continuing, define a fixed evaluation window or an explicit per-step criterion before you compare agents.

The boundary you choose is the foundation everything else stands on. Get it right, and your returns mean something. Get it wrong, and you'll spend hours debugging an agent that was never the problem.