Bellman equations, backups, fixed points, and recursive value reasoning used to connect immediate outcomes with continuation value.
Tagged articles
29 articles in this tag.
Set the discount factor to 1 in a continuing task and the return sum diverges. There is nothing finite left for a Bellman equation to solve for. Average…
Read tutorial
Your agent has no finish line. No terminal state ends the run, no episode boundary resets the world, and no natural horizon cuts the future off. You reach…
Read tutorial
A Bellman equation states a consistency condition. A Bellman backup is the mechanical step that enforces it. The difference between reading an equation and…
Read tutorial
Two guesses walk into the same operator. One is all zeros. One is absurdly large. If the Bellman operator really is a contraction, their distance should…
Read tutorial
The first time you see the Bellman equation, it looks like a symbol puzzle. Greek letters, nested expectations, a value function that seems to appear on…
Read tutorial
You can read $V(s) = R + \gamma V(s')$ a hundred times and still freeze when a variant shows up with a max, an expectation, or a $Q$ instead of a $V$. The…
Read tutorial
Everyone repeats that the deadly triad diverges. Almost nobody watches it happen. So let's build a predictor small enough to update by hand, then watch the…
Read tutorial
The scalar Bellman backup hands you a number. The distributional backup has to hand you a shape — and then bend that shape back onto a finite set of atoms.
Read tutorial
Two actions can offer the same expected return and still feel completely different to the agent that takes them. One delivers a steady, predictable…
Read tutorial
You can explain replay buffers and target networks. You have never watched a loss curve flatline while episode return refuses to move. That gap between…
Read tutorial
You know the SARSA target. You know the Q-learning target. Then someone says "Expected SARSA" and your brain stalls on one word: expectation. What action…
Read tutorial
A planner that looks accurate one step ahead can still produce value estimates that drift badly over a rollout. The usual reaction is to blame the model.…
Read tutorial
Your value network's loss curve flattens to something that looks like zero. The run looks healthy. Then you watch the greedy policy walk straight into a…
Read tutorial
Most reinforcement learning tutorials treat randomness as a ladder: you need it to explore, and once training converges, you kick it away. Maximum-entropy…
Read tutorial
A flat policy re-decides everything, every step. Options let it decide once and commit.
Read tutorial
That distinction matters. You can read about backups for an hour and still not trust them. The moment you watch a value estimate move — one number, one…
Read tutorial
You run a round of policy iteration. The greedy policy looks better on paper, so you deploy it. The measured return drops. Nothing in your code threw an…
Read tutorial
You write down the MDP Bellman equation, swap $s$ for $b$, and the result looks plausible. It also predicts nothing you can act on. The symbol changed; the…
Read tutorial
You can copy the Q-learning update into code and watch it work. Then someone asks why the target uses the best next action when the agent just took a…
Read tutorial
Most people start reinforcement learning the wrong way. They install a deep RL library, copy a DQN or PPO example, watch the agent fail—or worse, behave…
Read tutorial
A Bellman backup is a prediction. A robust Bellman backup is a prediction that has already survived an argument with an adversary. The difference between…
Read tutorial
A planner is only as stable as the model it trusts. Change one transition probability by a few points and watch the preferred action move.
Read tutorial
An option that runs for five steps collects five rewards. The primitive Bellman backup discounts one. That single mismatch is the whole reason the…
Read tutorial
Most actor-critic methods force you to choose between sample efficiency and stable exploration. Soft Actor-Critic (SAC) refuses the trade—and that refusal…
Read tutorial
You add an entropy bonus to your objective, and the whole value structure shifts under your feet. The max in the Bellman backup becomes a log-sum-exp. The…
Read tutorial
A Q-learning agent that has already found the goal still has to walk back there to learn from it again. Dyna-Q refuses that tax: it caches what happened…
Read tutorial
Your training loop ends every episode the same way. One flag flips, you reset, you move on. Somewhere in that single flag, two completely different events…
Read tutorial
You can recite that advantage equals Q minus V and still have no way to tell whether a program that prints those numbers is correct. That gap between…
Read tutorial
Value iteration is not a loop that guesses better. It is a machine that shrinks the distance between any two value estimates by a factor of gamma on every…
Read tutorial