advanced
11 min read
When Can an Action Shield Support a Safety Claim?
A shield that rejects unsafe actions every run can still support no safety claim at all.
Read tutorialmethod
Model-Based Reinforcement Learning
Learning or using transition and reward models for planning, including model error and the tradeoff between imagined and real experience.
Tagged articles
Model-Based Reinforcement Learning articles
14 articles in this tag.
intermediate
12 min read
Bellman Backups and Dynamic Programming for Reinforcement Learning
A Bellman equation states a consistency condition. A Bellman backup is the mechanical step that enforces it. The difference between reading an equation and…
Read tutorial
advanced
10 min read
Choosing a Planning Horizon in Model-Based RL
The agent plans beautifully for five steps. Then the imagined future quietly stops matching reality, and the only knob you reach for is horizon length.
Read tutorial
advanced
13 min read
Dyna-Style Reinforcement Learning: Mixing Real and Imagined Experience
Every real transition in a Dyna-style agent does two jobs at once: it teaches the value learner and it teaches the model. Then the model becomes a second…
Read tutorial
intermediate
12 min read
How Model Error Propagates Across a Finite Planning Horizon
A planner that looks accurate one step ahead can still produce value estimates that drift badly over a rollout. The usual reaction is to blame the model.…
Read tutorial
intermediate
10 min read
Model-Based vs Model-Free Reinforcement Learning: What Does the Agent Know?
The real difference between model-based and model-free reinforcement learning is not how much the agent learns. It is whether the agent can predict what…
Read tutorial
intermediate
9 min read
Compounding Model Error in Reinforcement Learning: Why Planning Drifts
A model that predicts the next state almost perfectly can still produce plans that drift into nonsense after a few imagined steps. The mistake is treating…
Read tutorial
advanced
11 min read
Model Predictive Control for Reinforcement Learning: Plan, Act, Replan
A ten-step plan is a promise your model cannot keep. Execute step one, measure what actually happened, and throw the other nine away.
Read tutorial
intermediate
10 min read
Test How Planning Horizon Changes Decisions Under Model Error
A planner can be confidently wrong. Give it a slightly inaccurate model and a long enough horizon, and it will walk off a cliff while reporting that the…
Read tutorial
intermediate
9 min read
Planning With a Learned Model in Reinforcement Learning
An agent that only learns from real experience is sample-hungry. An agent that trusts its own imagined experience can be confidently wrong. Planning with a…
Read tutorial
beginner
12 min read
Implement Policy Evaluation and Value Iteration for a Small MDP
That distinction matters. You can read about backups for an hour and still not trust them. The moment you watch a value estimate move — one number, one…
Read tutorial
advanced
9 min read
Robust MDPs: Planning With Transition Uncertainty Sets
A planner is only as stable as the model it trusts. Change one transition probability by a few points and watch the preferred action move.
Read tutorial
intermediate
7 min read
Implement Tabular Dyna-Q: Learn From Real and Simulated Transitions
A Q-learning agent that has already found the goal still has to walk back there to learn from it again. Dyna-Q refuses that tax: it caches what happened…
Read tutorial
beginner
9 min read
Transition Dynamics and Reward Models in Reinforcement Learning
You can name the parts of an MDP. States, actions, rewards, transitions—you have the vocabulary down. But if I ask you what the environment model actually…
Read tutorial