Choosing a Planning Horizon in Model-Based RL

The agent plans beautifully for five steps. Then the imagined future quietly stops matching reality, and the only knob you reach for is horizon length.

Here is the weak default: longer horizon equals more foresight. It is not wrong, exactly. It is just incomplete in a way that costs you weeks. Horizon buys foresight at the price of compounding model error and search cost. Replanning is what lets you buy foresight without paying the full error bill. The decision rule this article builds toward: set the horizon to the shortest depth that exposes the reward signal, measure error-versus-depth before trusting it, and buy additional foresight through replanning rather than rollout length.

What the Horizon Actually Controls

The horizon H is the number of imagined steps the planner optimizes over before committing an action. It is not the episode length. It is not the discount factor. Those are separate quantities, and conflating them is the first place horizon tuning goes wrong.

The second distinction matters more. An open-loop rollout of depth H commits to the entire imagined action sequence. A receding-horizon controller executes only the first action, then replans from the observed state. Same H, radically different error exposure. In the open-loop case, every step of model error accumulates into the plan you execute. In the receding-horizon case, error is truncated at the execution boundary—you commit to one step, then re-anchor to reality. But the planner still used the full H-step rollout to choose that one action, so prediction error inside the planning computation can still produce a bad first move.

Horizon is also not the same as search budget. Both cost compute, but they fail differently. A short horizon with a large search budget explores the same shallow tree thoroughly. A long horizon with a small search budget samples a deep tree sparsely and may miss the good branch entirely. When you tune one, hold the other fixed, or you will not know which one moved the result.

I assume you already know how a learned model produces imagined transitions and why one-step error compounds across depth. That mechanism is the floor this article stands on. What we need now is how to choose H given that floor.

The Three Costs That Move With H

Three quantities change as H grows, and they do not peak at the same place.

Foresight value rises with H until the planner can see the reward-relevant structure of the task. Once H covers that structure, additional depth adds nothing. A planner that can already see the goal does not benefit from imagining further past it.

Compounding model error rises monotonically with depth. The value estimate at the end of a long rollout is the least trustworthy part of the plan, because it has absorbed the most prediction error. This is the mechanism that makes long rollouts drift.

Compute and search difficulty also rise with H. Cost grows with depth, and gradient-free shooting optimizers increasingly hit local optima as credit assignment spreads across more steps. A planner that works at depth 5 may fail at depth 50 not because the model is worse but because the optimizer cannot find the good sequence.

The useful horizon sits where foresight has mostly saturated but error and cost have not yet dominated. That band is task-specific and model-specific. It is not a number you copy from a paper.

Common mistake: Treating H as a hyperparameter to grid-search blindly. You are searching a three-way tradeoff with a one-dimensional sweep, and you will find a local optimum that does not transfer.

Reading Model Accuracy and Uncertainty as Horizon Budget

Before you pick H, measure how much horizon your model can actually support.

Start with one-step prediction error on held-out real transitions. That number sets the floor for how fast multi-step error accumulates. If one-step error is already large, no horizon choice will save the planner. Fix the model or the representation first.

Then track error as a function of rollout depth. Roll the model forward k steps from real states, compare against what actually happened, and plot the error curve. The depth where error crosses your tolerance is an empirical ceiling. That ceiling is your model's horizon budget, and it is a measurement, not a guess.

The tolerance itself is task- and planner-specific. There is no universal prediction-error threshold that tells you when to stop. What matters is whether extra depth changes the action the planner selects. A state-prediction error that looks large in raw units may be irrelevant if it does not alter the ranking of candidate actions. Conversely, a small error in a reward-critical dimension can flip the decision. So measure two things: multi-step state and reward error to diagnose how the model degrades, and action-ranking agreement or realized-return gap to decide whether additional depth is still buying you anything useful.

Separate aleatoric noise from epistemic uncertainty while you are at it. Aleatoric noise is irreducible stochasticity in the environment; it caps accuracy but does not compound with depth in a way you can fix by planning shorter. Epistemic uncertainty is high in rarely visited regions, which means the horizon should shrink where the model is extrapolating. A planner that confidently extrapolates twenty steps into unseen state space is not planning. It is hallucinating with good posture.

Partial observability and unstable dynamics cap usable horizon regardless of model capacity. If the true state is hidden, the model cannot predict what it cannot see, and depth only amplifies the blind spot. The intervention differs by cause: stochasticity may favor risk-aware or expectation-aware planning, epistemic uncertainty favors conservative planning or more data, and partial observability favors belief-state augmentation. Do not treat uncertainty as a single scalar penalty on H.

The practical check: compare imagined returns against real returns at depth H. A widening gap is the signal to shorten, not to train longer.

Matching Horizon to Task Structure

The right H is a property of the task's reward and dynamics structure, not a universal constant.

If reward is dense and local, short horizons are sufficient and cheap. Long horizons add error for no foresight gain. If reward is sparse and only reachable after a long causal chain, the horizon must at least span the chain. Otherwise the planner never sees the signal it is optimizing for, and it optimizes nothing.

Temporally extended actions or skills reduce the effective task horizon. If a skill covers ten primitive steps, the same wall-clock foresight needs fewer model steps, and the model only has to predict the skill's outcome rather than every intermediate transition. This is why skill-based planning extends usable horizon without improving the underlying model.

One pattern worth understanding: very short and very long horizons can converge to similar performance in some environments for opposite reasons. The short horizon avoids error accumulation; the long horizon captures structure the short one misses. Do not read that as "horizon does not matter." It means two different mechanisms produced the same score, and they will diverge the moment the environment changes.

When to use a long H: sparse reward, low one-step error, cheap simulation. When not to: high-dimensional observations, unstable dynamics, tight compute.

Replanning Changes the Math

Horizon and replanning interval are two separate decisions. Most people collapse them into one and then wonder why tuning is confusing.

Receding-horizon control executes only the first action, then replans from the observed state. This truncates how long you commit to a single open-loop plan—you stop executing a stale imagined trajectory after one step and re-anchor to what actually happened. But replanning does not make the planning computation error-free. The planner still imagined H steps to choose that first action, and if those imagined steps were wrong, the chosen action can still be wrong. Replanning limits how far execution drifts from reality; it does not repair the prediction error inside each planning call.

A short horizon with frequent replanning can outperform a long horizon with rare replanning on the same model, because the short-horizon planner keeps correcting against reality while the long-horizon planner keeps trusting its imagination. But the horizon still matters under replanning. It determines whether the planner can see reward that lies beyond the immediate step. Replanning fixes execution drift; it does not fix myopia. A planner that replans every step but only looks one step ahead will still walk into a trap it could have seen from three steps back.

Replanning frequency is its own cost knob. More frequent replanning improves tracking accuracy and raises compute. The tradeoff is real, but it is a different tradeoff from horizon length.

Tip: Set H to the shortest depth that exposes the reward signal. Then set replanning frequency to the fastest rate your compute budget allows. These are two knobs, tuned in that order.

A Practical Procedure for Picking H

Here is the sequence I would run on a current agent.

Step 1. Measure one-step model error and error-versus-depth on held-out transitions. Plot the curve. This is your error ceiling.

Step 2. Find the minimum depth at which the planner's value estimate correlates with realized return. That is your foresight floor. Below it, the planner is guessing.

Step 3. Choose H inside the feasible interval: at or above the foresight floor, and at or below the lower of your error ceiling and your compute ceiling. If the floor sits above the ceiling, the interval is empty—stop tuning H and change something structural: the model or representation, the planner, the reward signal, the value bootstrap at the horizon edge, or the temporal abstraction.

Step 4. Within the feasible interval, prefer the smallest H that clears the foresight floor. Extra depth inside the interval buys marginal foresight at real error and compute cost. Only push toward the ceiling if you can show that deeper planning changes the selected action in a way that improves realized return.

Step 5. Fix H. Then tune replanning frequency separately. Do not move both at once.

Step 6. Re-check after model retraining. A better model raises the usable ceiling, and the horizon you chose last week may now be leaving foresight on the table.

The mistakes that cost the most time: tuning H before measuring model error, grid-searching H without a causal hypothesis, and changing H and replanning rate simultaneously so you cannot attribute the effect.

When Horizon Tuning Is the Wrong Fix

Horizon tuning is a narrow tool. It has boundaries, and crossing them wastes weeks.

If one-step error is already large, no horizon choice will save the planner. Fix the model or the representation first. If the task requires reasoning far beyond the model's accurate range, hierarchical or skill-level planning addresses the problem that horizon tuning cannot. If compute is the binding constraint, a better planner or a value-function bootstrap at the horizon edge may beat simply shortening H. If the environment is partially observable, adding state information can extend usable horizon more than any H setting.

State plainly what is known: error compounds with depth. What is inferred: your specific ceiling, from your specific error curve. What should not be assumed: that a published H transfers to your task. It was measured on a different model, a different environment, and a different compute budget.

The horizon is not a setting you inherit. It is a measurement you take.

Run the error-versus-depth measurement on your current model this week. Let that curve, not a paper's hyperparameter, set the ceiling. Then set H to the shortest depth that exposes the reward signal, and buy the rest of your foresight through replanning.