Runnable implementations and experiments intended for readers to execute, inspect, or modify as a hands-on learning activity.
Tagged articles
51 articles in this tag.
Your agent keeps taking an action you already know is wrong. You add a shield. Now you cannot tell whether it is helping or quietly starving the learner.
Read tutorial
Same opponent. Same six observed actions. Same code. Two different decisions — because the prior changed.
Read tutorial
A cloned policy can score 100% on the states it trained on and still walk off the path the moment it drives itself. That gap is the whole experiment.
Read tutorial
Two guesses walk into the same operator. One is all zeros. One is absurdly large. If the Bellman operator really is a contraction, their distance should…
Read tutorial
A learning algorithm that never improves is usually blamed for the wrong crime. More often, the environment quietly hands back the wrong next state, pays…
Read tutorial
A hand-picked penalty weight can look fine in one run and drift over budget in the next. The weight was never told what the budget is — so it can't respect…
Read tutorial
A greedy agent looks brilliant for the first fifty rounds. Then it stops learning, because it stopped asking questions.
Read tutorial
A CTDE run can post a beautiful training curve while the deployed policy quietly cheats. The critic saw the joint state. So did the actor. Nobody noticed,…
Read tutorial
You have the checklist memorized. State size. Data cost. Observability. Action structure. Baselines. Debugging burden. Then a real project lands on your…
Read tutorial
A policy cloned from demonstrations can look brilliant and still be useless. It scores well on the states it was shown, then stalls the moment the world…
Read tutorial
You tune gamma, the policy looks fine on your dashboard, and the task never ends. That is exactly when the scoreboard you optimized and the scoreboard you…
Read tutorial
You can explain replay buffers and target networks. You have never watched a loss curve flatline while episode return refuses to move. That gap between…
Read tutorial
A single episode is one sample. A single seed is one opinion. If you want to know whether your tabular agent learned anything, stop reading one number and…
Read tutorial
You add a curiosity bonus. Coverage climbs. The return curve goes flat. That gap is the whole lesson: a bonus is a second reward channel, and the only…
Read tutorial
A reward model can score 95% on the comparisons you trained it on and still be confidently wrong about what people meant. Here is the smallest honest…
Read tutorial
A GAE implementation can be wrong in a way that still prints plausible numbers. The terminal mask is where that happens.
Read tutorial
You can read PPO's clipped objective ten times and still not know whether your implementation works. The curve tells you. So does the clip fraction. So…
Read tutorial
You have read that options compress repeated action sequences into single decisions. Then you sit down to code one, and the boundary dissolves: where does…
Read tutorial
The Q-learning update looks like a ritual until you print the table before and after training and watch the numbers crawl toward the goal.
Read tutorial
Watch the two curves long enough and you will see the moment curiosity stops working for the agent and starts working for itself. The intrinsic bonus keeps…
Read tutorial
Two reward functions. One demonstrated path. No way to tell them apart from behavior alone.
Read tutorial
You understand what an MDP is. Then someone hands you a task description and a blank file, and the understanding evaporates.
Read tutorial
Two policies. One wins on average. The other wins on the days you actually have to survive. If your selection rule can't tell them apart, you're not…
Read tutorial
A team that coordinates perfectly with unlimited chatter can fall apart the moment you cap messages per step or flip a few bits. That collapse is the…
Read tutorial
Two agents, one shared reward, and a number that refuses to say who earned it. The fastest way to understand that refusal is to force it into arithmetic:…
Read tutorial
A return drop after an opponent switches is not evidence of a moving target. It is evidence that you have not yet measured the noise floor.
Read tutorial
Two estimators, one dataset, two different answers. That gap is not a bug in the math — it is the most useful signal you will get about whether your logs…
Read tutorial
An adaptive agent that beats a fixed opponent in one run can lose to the same opponent after a single behavior shift. The frequency estimate it trusts is a…
Read tutorial
A planner can be confidently wrong. Give it a slightly inaccurate model and a long enough horizon, and it will walk off a cliff while reporting that the…
Read tutorial
That distinction matters. You can read about backups for an hour and still not trust them. The moment you watch a value estimate move — one number, one…
Read tutorial
You have read the claim: subtract an action-independent baseline and the policy gradient keeps the same expected value while its variance drops. It is easy…
Read tutorial
That gap between a falling loss and a falling pole is the first thing to fix, because it tells you which number you are actually watching. In a…
Read tutorial
The policy-improvement theorem says a greedy step never hurts. Every real agent takes that step on estimates. This exercise finds the gap.
Read tutorial
You can recite the belief update formula and still have no idea whether you are applying it correctly. The way out is not more reading. It is a two-state…
Read tutorial
"Shaping speeds up learning without changing the optimal policy." That claim is true, and it is also the most misread sentence in reinforcement learning.…
Read tutorial
You collected two hundred comparisons. The reward model still ranks trajectories in an order nobody would defend. The instinct is to blame the model. Most…
Read tutorial
You have read that Q-learning overestimates. You nodded. You moved on. But you have never seen the bias as a number, so it stays a rumor.
Read tutorial
A memoryless agent can learn perfectly and still lose. It extracts every drop of signal from the observation it receives, then plateaus at a score a coin…
Read tutorial
Your training curve looks like a success story. The proxy reward climbs, the loss drops, and every dashboard agrees the agent is improving. Meanwhile the…
Read tutorial
Your policy scores well on the training environment. You change one parameter, run it again, and the score falls off a cliff. Is that a transfer failure,…
Read tutorial
A Bellman backup is a prediction. A robust Bellman backup is a prediction that has already survived an argument with an adversary. The difference between…
Read tutorial
Two agents train on the same grid, with the same rewards, the same learning rate, and the same epsilon. One learns to hug the edge of a penalty band. The…
Read tutorial
A self-play win rate can climb for a thousand iterations and still be lying to you. Here is the smallest instrument that catches it.
Read tutorial
Your policy scored well in simulation. On the target system, it behaves differently. All you have is two logs of the same episode — one from each system —…
Read tutorial
Your SAC implementation is probably correct. That is exactly why the failed run is so frustrating: the losses move, the entropy term behaves, and the agent…
Read tutorial
A Q-learning agent that has already found the goal still has to walk back there to learn from it again. Dyna-Q refuses that tax: it caches what happened…
Read tutorial
Watch a one-step TD learner on a short corridor. The reward sits at the far end, and the value of the first state crawls upward for hundreds of episodes.…
Read tutorial
A training loop can run clean, print a falling loss, and still learn the wrong thing. The bug is often one boolean.
Read tutorial
You bump clip_range from 0.2 to 0.3. The curve climbs higher. You conclude clipping matters.
Read tutorial
Two agents. Three actions each. Nine joint outcomes. If you cannot find the coordination failure in a table that small, you will not find it in a training…
Read tutorial
You can recite that advantage equals Q minus V and still have no way to tell whether a program that prints those numbers is correct. That gap between…
Read tutorial