Designing, measuring, comparing, and diagnosing RL experiments with repeatable procedures, suitable metrics, and uncertainty-aware interpretation.
Tagged articles
58 articles in this tag.
Your agent keeps taking an action you already know is wrong. You add a shield. Now you cannot tell whether it is helping or quietly starving the learner.
Read tutorial
A cloned policy can score 100% on the states it trained on and still walk off the path the moment it drives itself. That gap is the whole experiment.
Read tutorial
The agent plans beautifully for five steps. Then the imagined future quietly stops matching reality, and the only knob you reach for is horizon length.
Read tutorial
A hand-picked penalty weight can look fine in one run and drift over budget in the next. The weight was never told what the budget is — so it can't respect…
Read tutorial
A policy that earns more reward is not automatically a better policy. It may just have met easier contexts.
Read tutorial
A greedy agent looks brilliant for the first fifty rounds. Then it stops learning, because it stopped asking questions.
Read tutorial
A CTDE run can post a beautiful training curve while the deployed policy quietly cheats. The critic saw the joint state. So did the actor. Nobody noticed,…
Read tutorial
You train an offline agent on a fixed dataset. The final score looks strong. The learning curves climb. You feel the quiet satisfaction of a result that…
Read tutorial
A policy cloned from demonstrations can look brilliant and still be useless. It scores well on the states it was shown, then stalls the moment the world…
Read tutorial
You tune gamma, the policy looks fine on your dashboard, and the task never ends. That is exactly when the scoreboard you optimized and the scoreboard you…
Read tutorial
A single episode is one sample. A single seed is one opinion. If you want to know whether your tabular agent learned anything, stop reading one number and…
Read tutorial
A novelty bonus can make your agent look busy without making it look smart. The intrinsic-reward curve climbs, the state-visitation heatmap fills in, and…
Read tutorial
A policy that crushes its self-play twin can still collapse against a slightly different opponent. That single blind spot is why multi-agent reinforcement…
Read tutorial
You add a curiosity bonus. Coverage climbs. The return curve goes flat. That gap is the whole lesson: a bonus is a second reward channel, and the only…
Read tutorial
Imagine training an offline agent and watching its Q-values climb steadily. The critic grows more confident. The policy selects actions with increasing…
Read tutorial
A reward model can score 95% on the comparisons you trained it on and still be confidently wrong about what people meant. Here is the smallest honest…
Read tutorial
Your agent crushes its training level. Every episode, the same flawless path, the same high score, the same satisfying convergence curve. Then you move one…
Read tutorial
Your agent just nailed an episode. It reached the goal, collected a beautiful stream of rewards, and you felt that rush of validation. Then you rerun it,…
Read tutorial
You can read PPO's clipped objective ten times and still not know whether your implementation works. The curve tells you. So does the clip fraction. So…
Read tutorial
You have a dataset of logged trajectories collected under one policy, and you want to know what a different policy would have earned. The naive…
Read tutorial
Watch the two curves long enough and you will see the moment curiosity stops working for the agent and starts working for itself. The intrinsic bonus keeps…
Read tutorial
Two knobs. One update rule. If you are starting out in reinforcement learning, it is tempting to treat the learning rate and discount factor as twin dials…
Read tutorial
Two policies. One wins on average. The other wins on the days you actually have to survive. If your selection rule can't tell them apart, you're not…
Read tutorial
Two agents converge. Both policies are stable. Both agents are, by their own local test, optimal. The team still loses.
Read tutorial
A team that coordinates perfectly with unlimited chatter can fall apart the moment you cap messages per step or flip a few bits. That collapse is the…
Read tutorial
A policy that wins 90% of its self-play games can still lose to an opponent that never appeared in training.
Read tutorial
A return drop after an opponent switches is not evidence of a moving target. It is evidence that you have not yet measured the noise floor.
Read tutorial
Your training curves look beautiful. Reward is climbing, losses are shrinking, and the agent seems to have mastered its task. Then you deploy it against a…
Read tutorial
You have a log of episodes from one policy, a candidate policy you have never deployed, and a single number that claims to be the candidate's value. The…
Read tutorial
You have a new policy that looks promising. The next step feels obvious: deploy it, watch it interact with the real environment, and measure what happens.…
Read tutorial
A candidate policy that looks better on paper is not the same as a candidate policy that is better. Here is one small, honest rule for telling the…
Read tutorial
Two estimators, one dataset, two different answers. That gap is not a bug in the math — it is the most useful signal you will get about whether your logs…
Read tutorial
The moment you treat another learning agent as part of the environment, you are betting on a target that moves. Opponent modeling is the attempt to stop…
Read tutorial
An adaptive agent that beats a fixed opponent in one run can lose to the same opponent after a single behavior shift. The frequency estimate it trusts is a…
Read tutorial
A planner can be confidently wrong. Give it a slightly inaccurate model and a long enough horizon, and it will walk off a cliff while reporting that the…
Read tutorial
You have read the claim: subtract an action-independent baseline and the policy gradient keeps the same expected value while its variance drops. It is easy…
Read tutorial
That gap between a falling loss and a falling pole is the first thing to fix, because it tells you which number you are actually watching. In a…
Read tutorial
The policy-improvement theorem says a greedy step never hurts. Every real agent takes that step on estimates. This exercise finds the gap.
Read tutorial
"Shaping speeds up learning without changing the optimal policy." That claim is true, and it is also the most misread sentence in reinforcement learning.…
Read tutorial
You have read that Q-learning overestimates. You nodded. You moved on. But you have never seen the bias as a number, so it stays a rumor.
Read tutorial
A memoryless agent can learn perfectly and still lose. It extracts every drop of signal from the observation it receives, then plateaus at a score a coin…
Read tutorial
You changed two things at once. The agent got better. And now you have no idea which change earned the improvement.
Read tutorial
Your agent runs. The loop completes. No exceptions are raised. And yet the reward curve sits there like a flatline—or climbs promisingly for two hundred…
Read tutorial
Most people start reinforcement learning the wrong way. They install a deep RL library, copy a DQN or PPO example, watch the agent fail—or worse, behave…
Read tutorial
The reward curve climbs steadily. The training loss falls exactly as expected. And the outputs get visibly worse. Human reviewers start rejecting what the…
Read tutorial
Your training curve looks like a success story. The proxy reward climbs, the loss drops, and every dashboard agrees the agent is improving. Meanwhile the…
Read tutorial
Two policies can share the exact same expected return and still feel completely different to the person who has to live with the outcomes. One delivers…
Read tutorial
A policy scores well on a handful of held-out levels. The write-up says it "generalizes to unseen environments." Then deployment happens, and the policy…
Read tutorial
Your policy scores well on the training environment. You change one parameter, run it again, and the score falls off a cliff. Is that a transfer failure,…
Read tutorial
Two agents train on the same grid, with the same rewards, the same learning rate, and the same epsilon. One learns to hug the edge of a penalty band. The…
Read tutorial
A self-play win rate can climb for a thousand iterations and still be lying to you. Here is the smallest instrument that catches it.
Read tutorial
Your policy scored well in simulation. On the target system, it behaves differently. All you have is two logs of the same episode — one from each system —…
Read tutorial
A policy that scores perfectly in simulation is not a policy that works in the real world. It is a policy that solved the simulator.
Read tutorial
Your SAC implementation is probably correct. That is exactly why the failed run is so frustrating: the losses move, the entropy term behaves, and the agent…
Read tutorial
A Q-table makes the invisible visible. Before neural networks, before GPUs, before thousands of parameters, there is a simple grid, a handful of numbers,…
Read tutorial
Watch a one-step TD learner on a short corridor. The reward sits at the far end, and the value of the first state crawls upward for hundreds of episodes.…
Read tutorial
You bump clip_range from 0.2 to 0.3. The curve climbs higher. You conclude clipping matters.
Read tutorial
Two algorithms. Five seeds each. A forty-point gap in mean return. Someone asks which one to ship, and the honest answer is: you do not know yet. Not…
Read tutorial