Move from tables to learned policies with function approximation, policy gradients, actor-critic methods, PPO, preference learning, and deep RL.
Tutorials
Follow the learning path in order or jump directly to the RL concept or experiment you need.
When you hear "deep reinforcement learning," it's tempting to picture a neural network that somehow learned to think for itself. The network isn't doing…
Read tutorial
You've built a Q-learning agent. It works. For each state, you compare action values, pick the best one, and call that a policy. Then someone mentions…
Read tutorial
If you've worked through policy gradients, you've felt the problem: one lucky episode can make a mediocre action look brilliant, and the next update swings…
Read tutorial
The real question is not which family of algorithms is better. It is which one fits the structure of your problem—your action space, your data budget, and…
Read tutorial
If you have worked through policy gradients and deep RL, the language-model post-training landscape looks like a different discipline. RLHF. PPO. DPO.…
Read tutorial
Deep RL is not the destination of reinforcement learning. It is a specialized tool with a steep cost profile, and most learners reach for it before they…
Read tutorial
Tabular Q-learning converges reliably. Swap the table for a neural network, and the same update can oscillate, blow up, or quietly forget everything it…
Read tutorial
A policy network is not a bigger table. That is the mental model to dismantle before you touch policy-gradient code, because it quietly teaches you the…
Read tutorial
Your Q-table worked beautifully on the grid world. Every cell held a number, every update made sense, and you could read the agent's beliefs like an open…
Read tutorial
A policy-gradient update is a stride, not a leap. Take too long a step, and you can land somewhere worse than where you started—sometimes so much worse…
Read tutorial
You can write PPO's loss in three lines of code and still not know why the min() is there. That gap matters, because the min() is the whole algorithm.
Read tutorial
You bump clip_range from 0.2 to 0.3. The curve climbs higher. You conclude clipping matters.
Read tutorial
Hand-written rewards get hacked. Preference-based reinforcement learning replaces that fragile numeric signal with something that feels safer—human…
Read tutorial
A reward model is a compression. Thousands of noisy human comparisons get squeezed into one scalar function, and the compression itself is where meaning…
Read tutorial
More preference labels do not automatically mean a better reward model. The reward model is only as sharp as the comparisons that feed it.
Read tutorial
The reward curve climbs steadily. The training loss falls exactly as expected. And the outputs get visibly worse. Human reviewers start rejecting what the…
Read tutorial
A memoryless policy keeps repeating the same mistake because each observation hides the information it needs. The reflex is to "just add an RNN." The real…
Read tutorial
The REINFORCE update looks like a magic trick: multiply a log-probability by a return, add it to the parameters, and hope the policy improves. It is not a…
Read tutorial
That gap between a falling loss and a falling pole is the first thing to fix, because it tells you which number you are actually watching. In a…
Read tutorial
You can read PPO's clipped objective ten times and still not know whether your implementation works. The curve tells you. So does the clip fraction. So…
Read tutorial
A reward model can score 95% on the comparisons you trained it on and still be confidently wrong about what people meant. Here is the smallest honest…
Read tutorial
You can explain replay buffers and target networks. You have never watched a loss curve flatline while episode return refuses to move. That gap between…
Read tutorial
Two runs of the same policy-gradient code. Same environment, same seed, same learning rate. One climbs. One flatlines. The usual advice is "subtract a…
Read tutorial
Two reward models, same comparison data, different absolute scores—and both rank every trajectory identically. Neither is broken. The data never pinned…
Read tutorial
Your training curve looks like a success story. The proxy reward climbs, the loss drops, and every dashboard agrees the agent is improving. Meanwhile the…
Read tutorial
You have read the claim: subtract an action-independent baseline and the policy gradient keeps the same expected value while its variance drops. It is easy…
Read tutorial
A memoryless agent can learn perfectly and still lose. It extracts every drop of signal from the observation it receives, then plateaus at a score a coin…
Read tutorial
You collected two hundred comparisons. The reward model still ranks trajectories in an order nobody would defend. The instinct is to blame the model. Most…
Read tutorial
Your value network's loss curve flattens to something that looks like zero. The run looks healthy. Then you watch the greedy policy walk straight into a…
Read tutorial
You add prioritized replay. Training speeds up. Loss drops faster than it ever did with uniform sampling. Then, a few hundred thousand steps later, the…
Read tutorial
You have the checklist memorized. State size. Data cost. Observability. Action structure. Baselines. Debugging burden. Then a real project lands on your…
Read tutorial