Learning from sampled experience using bootstrapped estimates, including TD targets, multi-step returns, and eligibility traces.
Tagged articles
26 articles in this tag.
The agent makes a move. Nothing happens. It makes another move, then another. Twenty steps later, a reward arrives—and the agent has no idea which action…
Read tutorial
Everyone repeats that the deadly triad diverges. Almost nobody watches it happen. So let's build a predictor small enough to update by hand, then watch the…
Read tutorial
Your Q-learning agent trains beautifully for ten thousand steps. The loss curve dips, the reward trend climbs, and you start to believe the hard part is…
Read tutorial
Your Q-table has a favorite action. It picked that action because its estimate was highest, and now it is about to use that same estimate to decide how…
Read tutorial
You can explain replay buffers and target networks. You have never watched a loss curve flatline while episode return refuses to move. That gap between…
Read tutorial
You read one chapter that builds TD(λ) from a weighted mixture of n-step returns. You read another that updates a decaying trace and broadcasts the TD…
Read tutorial
One-step TD updates are beautifully local: take a step, observe the reward and next state, correct the value of the state you just left. Then you hit the…
Read tutorial
You know the SARSA target. You know the Q-learning target. Then someone says "Expected SARSA" and your brain stalls on one word: expectation. What action…
Read tutorial
A GAE implementation can be wrong in a way that still prints plausible numbers. The terminal mask is where that happens.
Read tutorial
A config file hands you lambda=0.95. You change it to 0.99, the run gets worse, you change it back, and you still cannot say what the number means. That is…
Read tutorial
Every actor-critic user hits the same wall shortly after their first working implementation: the advantage estimate looks fine in theory, but the policy…
Read tutorial
The Q-learning update looks like a ritual until you print the table before and after training and watch the numbers crawl toward the goal.
Read tutorial
Both methods learn from experience. One waits for the story to end before drawing conclusions. The other revises its belief after every sentence. The real…
Read tutorial
One-step TD updates fast but bets on a guess. Monte Carlo waits for the whole episode but trusts only what happened. The n-step return asks a sharper…
Read tutorial
You can recite the Q-learning update. Maybe you've even coded it in a toy grid world and watched the agent stumble toward a goal. But if someone asked you…
Read tutorial
Your agent's Q-values are climbing. Its actual performance is flatlining. And the gap between what the agent believes and what it achieves keeps widening.
Read tutorial
You can copy the Q-learning update into code and watch it work. Then someone asks why the target uses the best next action when the agent just took a…
Read tutorial
Two agents train on the same grid, with the same rewards, the same learning rate, and the same epsilon. One learns to hug the edge of a penalty band. The…
Read tutorial
Two agents start on the same grid. Same states, same actions, same rewards, same exploration schedule. One hugs the edge of a cliff and occasionally falls.…
Read tutorial
Tabular Q-learning converges reliably. Swap the table for a neural network, and the same update can oscillate, blow up, or quietly forget everything it…
Read tutorial
A Q-learning agent that has already found the goal still has to walk back there to learn from it again. Dyna-Q refuses that tax: it caches what happened…
Read tutorial
Your Q-table went quiet. The values stopped moving, the greedy policy stopped changing, and the learning curve flattened into something that looks like a…
Read tutorial
A Q-table makes the invisible visible. Before neural networks, before GPUs, before thousands of parameters, there is a simple grid, a handful of numbers,…
Read tutorial
Watch a one-step TD learner on a short corridor. The reward sits at the far end, and the value of the first state crawls upward for hundreds of episodes.…
Read tutorial
Your training loop ends every episode the same way. One flag flips, you reset, you move on. Somewhere in that single flag, two completely different events…
Read tutorial
A training loop can run clean, print a falling loss, and still learn the wrong thing. The bug is often one boolean.
Read tutorial