Designing, shaping, learning, or auditing reward signals and proxies, including the risks of objectives that diverge from intended outcomes.
Tagged articles
22 articles in this tag.
More preference labels do not automatically mean a better reward model. The reward model is only as sharp as the comparisons that feed it.
Read tutorial
A penalty is a price. A constraint is a wall. Most RL safety problems fail because practitioners build walls out of prices and then act surprised when the…
Read tutorial
The agent makes a move. Nothing happens. It makes another move, then another. Twenty steps later, a reward arrives—and the agent has no idea which action…
Read tutorial
Epsilon-greedy exploration is randomness without a reason. An exploration bonus is randomness with a budget and a purpose: it changes the reward signal…
Read tutorial
A reward model can score 95% on the comparisons you trained it on and still be confidently wrong about what people meant. Here is the smallest honest…
Read tutorial
Most beginners assume the algorithm is the hard part of reinforcement learning. Pick the right one, tune it well, and the agent will learn. Then they spend…
Read tutorial
Watch the two curves long enough and you will see the moment curiosity stops working for the agent and starts working for itself. The intrinsic bonus keeps…
Read tutorial
Curiosity is a loan of learning signal. It accelerates exploration when the borrowed interest points toward the goal—and it becomes a debt when the agent…
Read tutorial
Demonstrations tell you what an expert did. Inverse reinforcement learning asks why—and the answer is never uniquely determined by the behavior itself.
Read tutorial
Two reward functions. One demonstrated path. No way to tell them apart from behavior alone.
Read tutorial
Two reward models, same comparison data, different absolute scores—and both rank every trajectory identically. Neither is broken. The data never pinned…
Read tutorial
A shaped agent learns faster, the curve looks great, and the final policy is subtly wrong. The shaping term did not speed up learning; it quietly rewrote…
Read tutorial
"Shaping speeds up learning without changing the optimal policy." That claim is true, and it is also the most misread sentence in reinforcement learning.…
Read tutorial
Hand-written rewards get hacked. Preference-based reinforcement learning replaces that fragile numeric signal with something that feels safer—human…
Read tutorial
You collected two hundred comparisons. The reward model still ranks trajectories in an order nobody would defend. The instinct is to blame the model. Most…
Read tutorial
Your agent runs. The loop completes. No exceptions are raised. And yet the reward curve sits there like a flatline—or climbs promisingly for two hundred…
Read tutorial
If you have worked through policy gradients and deep RL, the language-model post-training landscape looks like a different discipline. RLHF. PPO. DPO.…
Read tutorial
The reward curve climbs steadily. The training loss falls exactly as expected. And the outputs get visibly worse. Human reviewers start rejecting what the…
Read tutorial
A reward model is a compression. Thousands of noisy human comparisons get squeezed into one scalar function, and the compression itself is where meaning…
Read tutorial
Your training curve looks like a success story. The proxy reward climbs, the loss drops, and every dashboard agrees the agent is improving. Meanwhile the…
Read tutorial
Your agent is doing one of two things: nothing, or the wrong thing. When rewards never arrive, it wanders forever without learning. When rewards arrive too…
Read tutorial
A reward is feedback. The return is the scoreboard. Beginners often treat them as the same thing—and then wonder why their agent grabs the pebble instead…
Read tutorial