Model Predictive Control for Reinforcement Learning: Plan, Act, Replan

A ten-step plan is a promise your model cannot keep. Execute step one, measure what actually happened, and throw the other nine away.

That single habit — plan far, commit shallow — is the whole of model predictive control. It is also the cleanest answer to a problem you already know from learned-model planning: imagined rollouts drift, and the further you look, the less you should believe. MPC does not try to fix the drift. It refuses to bet on the part of the plan where drift lives.

Here is the mental model I want you to carry through this article. The plan is disposable. The feedback is the asset. And the horizon is a bet on how far your model stays trustworthy.

The Plan You Throw Away

At every decision point, a model predictive controller solves an optimization problem over a finite prediction horizon. It rolls the model forward, scores candidate action sequences by predicted return, and picks the best one. Then it executes exactly one action — the first — and discards the rest.

That discard is the part that surprises people arriving from trajectory optimization. If you have ever computed a full optimal trajectory offline and then tracked it, your instinct is to queue the remaining actions and follow them. MPC does the opposite. The remaining actions are not a schedule. They are scaffolding for a decision you are about to make once and abandon.

Why abandon them? Because the next decision point re-solves from a freshly measured state. The new plan is anchored to what the environment actually did, not to what the previous plan predicted it would do. You never execute the tenth step of a plan whose first nine steps were imagined.

We already covered how a model is learned from real transitions and why imagined rollouts drift. This article is about the other half: how a controller consumes that model repeatedly instead of once. The model does not change. The consumption pattern does.

Why Replanning Is Error Insurance

Model error compounds across imagined steps. A small one-step bias does not stay small — it feeds into the next prediction, which feeds into the next, and the tail of a long open-loop plan is the least trustworthy part of it. This is the mechanism behind planning drift, and it is worth being precise about what replanning does and does not buy you.

Replanning converts an open-loop commitment into a closed-loop correction. It stops the controller from executing the unobserved tail of an old prediction, and it corrects state-estimation mismatch at the next decision point. You execute the first step of a plan whose first step starts from a measured state.

What replanning cannot do is erase error. It cannot remove a systematically biased model, undo damage the environment already incurred, or recover information the state never contained. If a disturbance pushes you off course, replanning reacts at the next measurement — it does not prevent the push. If your sensor lags, the "fresh" state is already stale. Replanning limits the damage from drift. It does not neutralize error.

The horizon length is the dial that controls how much you lean on the model. A long horizon exploits a good model — it lets the controller reason about consequences far ahead. A short horizon survives a bad one — it barely trusts the model at all. Choosing a horizon is choosing how far you believe your model.

Two boundaries matter here, and both are easy to miss.

Common mistake: Replanning does not fix a systematically biased model. If the model is wrong in the same direction every step, feedback keeps re-injecting the same wrong belief. Replanning limits damage from drift — error that accumulates — not from bias — error that repeats. A biased model will produce a controller that confidently walks into the same wall from a new starting point each time.

The second boundary is cost. Replanning buys robustness with compute, not for free. Every decision point requires a fresh solve. If the solve is expensive, the robustness is expensive, and there is a real deadline you can miss.

The Loop, Step by Step

Here is the workflow you can replay on any environment. It is short enough to hold in your head, which is exactly why it is worth memorizing.

1. Observe. Read the current state from the real environment. Not from the model's last prediction — from the environment.
2. Plan. Roll the model forward over the horizon and search or optimize for the action sequence that maximizes predicted return.
3. Act. Execute only the first action of that sequence.
4. Observe again. The environment returns the actual next state and reward. They may differ from what the plan predicted.
5. Replan. Discard the old sequence entirely and solve again from the observed state.

Then loop. The plan window slides forward one step at a time, always starting from a measured state, always committing to one action.

Picture a horizontal timeline with plan windows drawn as overlapping brackets. Each bracket spans the horizon. Each bracket advances by exactly one step. The executed action sits at the left edge of each bracket; the shaded tail to its right is the part you throw away. The overlap between consecutive brackets is the feedback: the region where the new plan replaces the old prediction with a measurement.

That sliding window is what "receding horizon" means. The horizon does not shrink. It recedes — it moves forward with you, always the same length ahead, always re-anchored.

Where the Value Function Enters

A finite horizon has a problem at its edge. Past the last planned step, the controller is blind. It will happily choose a sequence that looks excellent for ten steps and walks off a cliff on step eleven, because nothing in the objective penalizes step eleven.

The fix is a terminal estimate. A learned value function supplies it: instead of scoring the plan only by rewards collected inside the horizon, you add the predicted value of the state you land in at the horizon boundary. That turns a truncated plan into an approximate infinite-horizon decision.

This is the same structure you met in Bellman reasoning — one-step reward plus a discounted continuation prediction — just unrolled a fixed number of times and closed with a learned prediction instead of another backup. The Bellman relationship did not change. You stopped unrolling and asked the value function to finish the sentence.

The quality of that terminal estimate sets a ceiling on plan quality. A bad value function at the horizon makes long horizons worse, not better, because the controller now trusts a bad prediction at exactly the point where it has the least information. If your long-horizon plans behave strangely, check the terminal estimate before you blame the model.

What Learning Actually Does Here

The loop above stays the same no matter how you wire it. What changes is which components are fixed and which are learned. "MPC for reinforcement learning" is not one architecture — it is a set of answers to three independent questions.

Who selects the action? Either the optimizer selects it directly, or a learned policy selects it and MPC plays a supporting role. The common case is the optimizer selecting.

Where does the model come from? Either you have a known dynamics model, or you learn one from real transitions. A learned model is what makes this a model-based RL system rather than classical control.

What does learning supply? This is the axis people collapse too quickly. Learning can supply the model, the terminal value estimate, the controller's objective or constraints, or nothing at all.

Those axes combine freely. A controller can select actions via optimization while a learned value function supplies its terminal cost — that is still MPC as the decision procedure, and learning is doing real work. A controller can use a known model while RL tunes its objective weights from measured outcomes. Both are legitimate, and neither fits a clean two-box taxonomy.

The decision rule I use: add learning only where specification fails. If you have a trustworthy model and a well-posed objective, the optimizer alone is enough — do not add a learning loop you do not need. If the objective or constraints are hard to write down but you can measure outcomes, let RL tune them. If the model is the weak link, learn the model. Each learned component earns its place by covering a gap you cannot close by hand.

When Replanning Is the Wrong Tool

Receding-horizon planning is not a default. It is a choice with a cost, and there are cases where it is overkill or actively harmful.

• Cheap, fast, well-understood dynamics. A fixed policy or a one-step rule beats a per-step optimization you now have to maintain. If the dynamics are simple, the solve is pure overhead.
• Very expensive or slow decisions. If you cannot re-solve before the next decision is due, the loop degrades into open-loop control with extra latency. You paid for feedback and received delay.
• Hard safety constraints that must hold between replans. Replanning alone gives no guarantee. The gap between solves is where violations live, and a constraint that must hold continuously is not protected by a controller that only checks at decision points.
• Non-Markov observations. If the measured state does not summarize what matters, replanning from it re-injects the same missing information every step. Feedback cannot repair a state that never contained the answer.

And one more, which is less a boundary than a priority: when the model is the bottleneck, improving the model usually beats tuning the horizon. Replanning limits the damage from a mediocre model. It does not make a bad model good.

Tuning the Horizon Without Guessing

Horizon length is the one hyperparameter that most directly encodes your trust in the model, so do not pick it by feel. Measure it.

Sweep horizon length and plot real-environment return against horizon. The curve usually rises, plateaus, then falls as model error dominates the tail. That shape is the whole story: short horizons underuse the model, long horizons overtrust it, and the plateau is where the two forces balance.

Compare against two baselines. A one-step-greedy baseline tells you what you get with almost no planning. An open-loop baseline that commits to the entire plan tells you what you get with no feedback. The gap between the receding-horizon curve and those baselines isolates how much feedback is actually buying you — and if the gap is small, you are paying solve cost for nothing.

Watch solve time per step, not just return. A horizon that wins on return but misses the control deadline is not a win. The deadline is a hard constraint; return is a soft one.

Tip: Treat the curve as a diagnostic clue, not a verdict. A collapse only at long horizons is consistent with rollout drift — the tail is untrustworthy and replanning is doing its job. A collapse at all horizons points somewhere else: model bias, a bad terminal estimate, or an optimizer that is not actually finding good plans. To separate them, hold the terminal estimate and optimizer fixed and check short-horizon prediction error directly. The sweep narrows the suspects; it does not name the culprit alone.

This is the smallest observable skill to drill before building anything larger: one environment, three horizons, one plot. Run it before you touch the architecture.

The Horizon Is a Statement About Trust

The decision rule I want you to keep is this. The horizon is a statement about how far you trust your model. Replanning is how you keep that statement from becoming a commitment. Plan far enough to reason about consequences, commit shallow enough to survive being wrong, and let each new measurement correct the prediction before the old one does damage.

Take one environment you already have a model for. Run the three-horizon sweep. Read the curve before you touch anything else — before you tune the terminal estimate, before you change the model, before you add a learning loop. The curve will tell you whether your problem looks like drift, bias, or a horizon that was never a bet worth making. Then fix the component the evidence points at, not the one that is easiest to tune.