State values, action values, advantages, and related predictions of future outcomes used to compare decisions.
Tagged articles
62 articles in this tag.
If you've worked through policy gradients, you've felt the problem: one lucky episode can make a mediocre action look brilliant, and the next update swings…
Read tutorial
Set the discount factor to 1 in a continuing task and the return sum diverges. There is nothing finite left for a Bellman equation to solve for. Average…
Read tutorial
Your agent has no finish line. No terminal state ends the run, no episode boundary resets the world, and no natural horizon cuts the future off. You reach…
Read tutorial
A Bellman equation states a consistency condition. A Bellman backup is the mechanical step that enforces it. The difference between reading an equation and…
Read tutorial
Two guesses walk into the same operator. One is all zeros. One is absurdly large. If the Bellman operator really is a contraction, their distance should…
Read tutorial
The first time you see the Bellman equation, it looks like a symbol puzzle. Greek letters, nested expectations, a value function that seems to appear on…
Read tutorial
You can read $V(s) = R + \gamma V(s')$ a hundred times and still freeze when a variant shows up with a max, an expectation, or a $Q$ instead of a $V$. The…
Read tutorial
A team wins. The shared reward is +1. Every agent's gradient gets the same number, so the agent who stood still learns exactly as much as the agent who…
Read tutorial
Your team just won. The shared reward arrives, positive and unambiguous. Every agent updates its policy as if the victory were its own doing—and that is…
Read tutorial
Everyone repeats that the deadly triad diverges. Almost nobody watches it happen. So let's build a predictor small enough to update by hand, then watch the…
Read tutorial
Your Q-learning agent trains beautifully for ten thousand steps. The loss curve dips, the reward trend climbs, and you start to believe the hard part is…
Read tutorial
The scalar Bellman backup hands you a number. The distributional backup has to hand you a shape — and then bend that shape back onto a finite set of atoms.
Read tutorial
Two actions can offer the same expected return and still feel completely different to the agent that takes them. One delivers a steady, predictable…
Read tutorial
Your Q-table has a favorite action. It picked that action because its estimate was highest, and now it is about to use that same estimate to decide how…
Read tutorial
You can explain replay buffers and target networks. You have never watched a loss curve flatline while episode return refuses to move. That gap between…
Read tutorial
Every real transition in a Dyna-style agent does two jobs at once: it teaches the value learner and it teaches the model. Then the model becomes a second…
Read tutorial
You know the SARSA target. You know the Q-learning target. Then someone says "Expected SARSA" and your brain stalls on one word: expectation. What action…
Read tutorial
Imagine training an offline agent and watching its Q-values climb steadily. The critic grows more confident. The policy selects actions with increasing…
Read tutorial
A planner that looks accurate one step ahead can still produce value estimates that drift badly over a rollout. The usual reaction is to blame the model.…
Read tutorial
Your value network's loss curve flattens to something that looks like zero. The run looks healthy. Then you watch the greedy policy walk straight into a…
Read tutorial
A GAE implementation can be wrong in a way that still prints plausible numbers. The terminal mask is where that happens.
Read tutorial
A config file hands you lambda=0.95. You change it to 0.99, the run gets worse, you change it back, and you still cannot say what the number means. That is…
Read tutorial
Every actor-critic user hits the same wall shortly after their first working implementation: the advantage estimate looks fine in theory, but the policy…
Read tutorial
The Q-learning update looks like a ritual until you print the table before and after training and watch the numbers crawl toward the goal.
Read tutorial
Merging states is not compression. It is a claim about the future, and most merges quietly change the problem.
Read tutorial
A ten-step plan is a promise your model cannot keep. Execute step one, measure what actually happened, and throw the other nine away.
Read tutorial
A team where every agent picks its own best action can still lose. Monotonic value factorization is the constraint that rules out one specific failure —…
Read tutorial
Both methods learn from experience. One waits for the story to end before drawing conclusions. The other revises its belief after every sentence. The real…
Read tutorial
One-step TD updates fast but bets on a guess. Monte Carlo waits for the whole episode but trusts only what happened. The n-step return asks a sharper…
Read tutorial
An offline RL agent can report a sky-high value for an action it never actually tried. The critic is extrapolating into territory the dataset never…
Read tutorial
An agent that has seen an action once does not know its value. It knows one sample. Optimism is the discipline of treating thin evidence as a reason to…
Read tutorial
That distinction matters. You can read about backups for an hour and still not trust them. The moment you watch a value estimate move — one number, one…
Read tutorial
Two runs of the same policy-gradient code. Same environment, same seed, same learning rate. One climbs. One flatlines. The usual advice is "subtract a…
Read tutorial
You can compute value estimates. You can trace a Bellman backup through a small grid. But if someone asked you to explain how those numbers actually change…
Read tutorial
The policy-improvement theorem says a greedy step never hurts. Every real agent takes that step on estimates. This exercise finds the gap.
Read tutorial
You run a round of policy iteration. The greedy policy looks better on paper, so you deploy it. The measured return drops. Nothing in your code threw an…
Read tutorial
You write down the MDP Bellman equation, swap $s$ for $b$, and the result looks plausible. It also predicts nothing you can act on. The symbol changed; the…
Read tutorial
Scoring a policy and improving a policy feel like the same job. They are not. One is a measurement. The other is a search. Confusing them is one of the…
Read tutorial
You can recite the Q-learning update. Maybe you've even coded it in a toy grid world and watched the agent stumble toward a goal. But if someone asked you…
Read tutorial
You have read that Q-learning overestimates. You nodded. You moved on. But you have never seen the bias as a number, so it stays a rumor.
Read tutorial
Your agent's Q-values are climbing. Its actual performance is flatlining. And the gap between what the agent believes and what it achieves keeps widening.
Read tutorial
You can copy the Q-learning update into code and watch it work. Then someone asks why the target uses the best next action when the agent just took a…
Read tutorial
Most people start reinforcement learning the wrong way. They install a deep RL library, copy a DQN or PPO example, watch the agent fail—or worse, behave…
Read tutorial
You're halfway through a tutorial when a sentence stops you cold: "The agent updates its Q-value using the discounted return from the trajectory." You know…
Read tutorial
A Bellman backup is a prediction. A robust Bellman backup is a prediction that has already survived an argument with an adversary. The difference between…
Read tutorial
A planner is only as stable as the model it trusts. Change one transition probability by a few points and watch the preferred action move.
Read tutorial
Two agents start on the same grid. Same states, same actions, same rewards, same exploration schedule. One hugs the edge of a cliff and occasionally falls.…
Read tutorial
An option that runs for five steps collects five rewards. The primitive Bellman backup discounts one. That single mismatch is the whole reason the…
Read tutorial
Most actor-critic methods force you to choose between sample efficiency and stable exploration. Soft Actor-Critic (SAC) refuses the trade—and that refusal…
Read tutorial
You add an entropy bonus to your objective, and the whole value structure shifts under your feet. The max in the Bellman backup becomes a log-sum-exp. The…
Read tutorial
Tabular Q-learning converges reliably. Swap the table for a neural network, and the same update can oscillate, blow up, or quietly forget everything it…
Read tutorial
A Q-table makes the invisible visible. Before neural networks, before GPUs, before thousands of parameters, there is a simple grid, a handful of numbers,…
Read tutorial
Your Q-table worked beautifully on the grid world. Every cell held a number, every update made sense, and you could read the agent's beliefs like an open…
Read tutorial
Watch a one-step TD learner on a short corridor. The reward sits at the far end, and the value of the first state crawls upward for hundreds of episodes.…
Read tutorial
DDPG doesn't fail because its policy is weak. It fails because its critic quietly starts lying, and the policy trusts every word.
Read tutorial
The real question is not which family of algorithms is better. It is which one fits the structure of your problem—your action space, your data budget, and…
Read tutorial
A team shares one reward, but at execution time each agent must pick its own action from only its own observation. Value decomposition answers that…
Read tutorial
Two agents. Three actions each. Nine joint outcomes. If you cannot find the coordination failure in a table that small, you will not find it in a training…
Read tutorial
A reward tells you what just happened. A value function predicts the return you can expect from here to the end of the episode—and that forward-looking…
Read tutorial
You can recite that advantage equals Q minus V and still have no way to tell whether a program that prints those numbers is correct. That gap between…
Read tutorial
You're reading an actor-critic explanation and you hit three symbols: V, Q, and A. The critic estimates V. The policy gradient weights updates by A.…
Read tutorial
Value iteration is not a loop that guesses better. It is a machine that shrinks the distance between any two value estimates by a factor of gamma on every…
Read tutorial