Turn experience into expectations about the future with values, Q-values, advantage, Bellman updates, models, planning, and policy improvement.
Tutorials
Follow the learning path in order or jump directly to the RL concept or experiment you need.
A reward tells you what just happened. A value function predicts the return you can expect from here to the end of the episode—and that forward-looking…
Read tutorial
The first time you see the Bellman equation, it looks like a symbol puzzle. Greek letters, nested expectations, a value function that seems to appear on…
Read tutorial
You can read $V(s) = R + \gamma V(s')$ a hundred times and still freeze when a variant shows up with a max, an expectation, or a $Q$ instead of a $V$. The…
Read tutorial
That distinction matters. You can read about backups for an hour and still not trust them. The moment you watch a value estimate move — one number, one…
Read tutorial
You're reading an actor-critic explanation and you hit three symbols: V, Q, and A. The critic estimates V. The policy gradient weights updates by A.…
Read tutorial
The real difference between model-based and model-free reinforcement learning is not how much the agent learns. It is whether the agent can predict what…
Read tutorial
The agent-environment loop feels simple until you try to make an algorithm learn. Then the loop stops being enough. The agent needs to know what it can…
Read tutorial
A Bellman equation states a consistency condition. A Bellman backup is the mechanical step that enforces it. The difference between reading an equation and…
Read tutorial
Scoring a policy and improving a policy feel like the same job. They are not. One is a measurement. The other is a search. Confusing them is one of the…
Read tutorial
An agent that only learns from real experience is sample-hungry. An agent that trusts its own imagined experience can be confidently wrong. Planning with a…
Read tutorial
You can name the parts of an MDP. States, actions, rewards, transitions—you have the vocabulary down. But if I ask you what the environment model actually…
Read tutorial
Value iteration is not a loop that guesses better. It is a machine that shrinks the distance between any two value estimates by a factor of gamma on every…
Read tutorial
You can compute value estimates. You can trace a Bellman backup through a small grid. But if someone asked you to explain how those numbers actually change…
Read tutorial
A model that predicts the next state almost perfectly can still produce plans that drift into nonsense after a few imagined steps. The mistake is treating…
Read tutorial
Two actions can offer the same expected return and still feel completely different to the agent that takes them. One delivers a steady, predictable…
Read tutorial
Every real transition in a Dyna-style agent does two jobs at once: it teaches the value learner and it teaches the model. Then the model becomes a second…
Read tutorial
A policy that acts on the latest observation alone is guessing. A belief state is how you stop guessing.
Read tutorial
The agent never sees the state. It sees a flicker of evidence, acts, and sees another flicker. Yet the entire decision problem collapses into a fixed-size…
Read tutorial
You can recite the belief update formula and still have no idea whether you are applying it correctly. The way out is not more reading. It is a two-state…
Read tutorial
A flat policy re-decides everything, every step. Options let it decide once and commit.
Read tutorial
An option that runs for five steps collects five rewards. The primitive Bellman backup discounts one. That single mismatch is the whole reason the…
Read tutorial
You have read that options compress repeated action sequences into single decisions. Then you sit down to code one, and the boundary dissolves: where does…
Read tutorial
A flat agent that learns the easy part of a long task and then stalls is not asking for a hierarchy. It is asking for a diagnosis.
Read tutorial
Your agent has no finish line. No terminal state ends the run, no episode boundary resets the world, and no natural horizon cuts the future off. You reach…
Read tutorial
Set the discount factor to 1 in a continuing task and the return sum diverges. There is nothing finite left for a Bellman equation to solve for. Average…
Read tutorial
You tune gamma, the policy looks fine on your dashboard, and the task never ends. That is exactly when the scoreboard you optimized and the scoreboard you…
Read tutorial
A ten-step plan is a promise your model cannot keep. Execute step one, measure what actually happened, and throw the other nine away.
Read tutorial
The agent plans beautifully for five steps. Then the imagined future quietly stops matching reality, and the only knob you reach for is horizon length.
Read tutorial
The scalar Bellman backup hands you a number. The distributional backup has to hand you a shape — and then bend that shape back onto a finite set of atoms.
Read tutorial
You run a round of policy iteration. The greedy policy looks better on paper, so you deploy it. The measured return drops. Nothing in your code threw an…
Read tutorial
A planner that looks accurate one step ahead can still produce value estimates that drift badly over a rollout. The usual reaction is to blame the model.…
Read tutorial
A Q-learning agent that has already found the goal still has to walk back there to learn from it again. Dyna-Q refuses that tax: it caches what happened…
Read tutorial
A planner can be confidently wrong. Give it a slightly inaccurate model and a long enough horizon, and it will walk off a cliff while reporting that the…
Read tutorial
You write down the MDP Bellman equation, swap $s$ for $b$, and the result looks plausible. It also predicts nothing you can act on. The symbol changed; the…
Read tutorial
Two guesses walk into the same operator. One is all zeros. One is absurdly large. If the Bellman operator really is a contraction, their distance should…
Read tutorial
Two situations look identical to your agent. It takes the same action. One time the world rewards it; the next time the world punishes it. You blame the…
Read tutorial
A planner is only as stable as the model it trusts. Change one transition probability by a few points and watch the preferred action move.
Read tutorial
Merging states is not compression. It is a claim about the future, and most merges quietly change the problem.
Read tutorial
The policy-improvement theorem says a greedy step never hurts. Every real agent takes that step on estimates. This exercise finds the gap.
Read tutorial
A Bellman backup is a prediction. A robust Bellman backup is a prediction that has already survived an argument with an adversary. The difference between…
Read tutorial
You understand what an MDP is. Then someone hands you a task description and a blank file, and the understanding evaporates.
Read tutorial
You can recite that advantage equals Q minus V and still have no way to tell whether a program that prints those numbers is correct. That gap between…
Read tutorial