Policies as decision rules and the process of evaluating, improving, or optimizing behavior.
Tagged articles
52 articles in this tag.
A Bellman equation states a consistency condition. A Bellman backup is the mechanical step that enforces it. The difference between reading an equation and…
Read tutorial
The first time you see the Bellman equation, it looks like a symbol puzzle. Greek letters, nested expectations, a value function that seems to appear on…
Read tutorial
You can read $V(s) = R + \gamma V(s')$ a hundred times and still freeze when a variant shows up with a max, an expectation, or a $Q$ instead of a $V$. The…
Read tutorial
The agent plans beautifully for five steps. Then the imagined future quietly stops matching reality, and the only knob you reach for is horizon length.
Read tutorial
You move from discrete actions to continuous control, wire up a Gaussian policy head, and the agent either jitters uselessly around a target or slams…
Read tutorial
Your team scores well in the training harness. You strip the centralized critic, deploy the actors, and the score collapses — even though the environment…
Read tutorial
A stochastic policy averages over actions. A deterministic policy climbs the critic's slope at the single action it outputs. That swap—from averaging to…
Read tutorial
A policy that crushes its self-play twin can still collapse against a slightly different opponent. That single blind spot is why multi-agent reinforcement…
Read tutorial
You know the SARSA target. You know the Q-learning target. Then someone says "Expected SARSA" and your brain stalls on one word: expectation. What action…
Read tutorial
You have a dataset of logged trajectories collected under one policy, and you want to know what a different policy would have earned. The naive…
Read tutorial
Most reinforcement learning tutorials treat randomness as a ladder: you need it to explore, and once training converges, you kick it away. Maximum-entropy…
Read tutorial
Two policies. One wins on average. The other wins on the days you actually have to survive. If your selection rule can't tell them apart, you're not…
Read tutorial
A ten-step plan is a promise your model cannot keep. Execute step one, measure what actually happened, and throw the other nine away.
Read tutorial
Two agents converge. Both policies are stable. Both agents are, by their own local test, optimal. The team still loses.
Read tutorial
You have a log of episodes from one policy, a candidate policy you have never deployed, and a single number that claims to be the candidate's value. The…
Read tutorial
You have a new policy that looks promising. The next step feels obvious: deploy it, watch it interact with the real environment, and measure what happens.…
Read tutorial
A candidate policy that looks better on paper is not the same as a candidate policy that is better. Here is one small, honest rule for telling the…
Read tutorial
Two estimators, one dataset, two different answers. That gap is not a bug in the math — it is the most useful signal you will get about whether your logs…
Read tutorial
A flat policy re-decides everything, every step. Options let it decide once and commit.
Read tutorial
A policy network is not a bigger table. That is the mental model to dismantle before you touch policy-gradient code, because it quietly teaches you the…
Read tutorial
A policy is not a rulebook. It is a decision rule that keeps producing choices as the agent moves through the world—and every choice reshapes which futures…
Read tutorial
That distinction matters. You can read about backups for an hour and still not trust them. The moment you watch a value estimate move — one number, one…
Read tutorial
You've built a Q-learning agent. It works. For each state, you compare action values, pick the best one, and call that a policy. Then someone mentions…
Read tutorial
You can compute value estimates. You can trace a Bellman backup through a small grid. But if someone asked you to explain how those numbers actually change…
Read tutorial
The policy-improvement theorem says a greedy step never hurts. Every real agent takes that step on estimates. This exercise finds the gap.
Read tutorial
You run a round of policy iteration. The greedy policy looks better on paper, so you deploy it. The measured return drops. Nothing in your code threw an…
Read tutorial
You write down the MDP Bellman equation, swap $s$ for $b$, and the result looks plausible. It also predicts nothing you can act on. The symbol changed; the…
Read tutorial
Scoring a policy and improving a policy feel like the same job. They are not. One is a measurement. The other is a search. Confusing them is one of the…
Read tutorial
A policy-gradient update is a stride, not a leap. Take too long a step, and you can land somewhere worse than where you started—sometimes so much worse…
Read tutorial
You can recite the Q-learning update. Maybe you've even coded it in a toy grid world and watched the agent stumble toward a goal. But if someone asked you…
Read tutorial
You can copy the Q-learning update into code and watch it work. Then someone asks why the target uses the best next action when the agent just took a…
Read tutorial
A memoryless policy keeps repeating the same mistake because each observation hides the information it needs. The reflex is to "just add an RNN." The real…
Read tutorial
A memoryless agent can learn perfectly and still lose. It extracts every drop of signal from the observation it receives, then plateaus at a score a coin…
Read tutorial
If you have worked through policy gradients and deep RL, the language-model post-training landscape looks like a different discipline. RLHF. PPO. DPO.…
Read tutorial
Most people start reinforcement learning the wrong way. They install a deep RL library, copy a DQN or PPO example, watch the agent fail—or worse, behave…
Read tutorial
You're halfway through a tutorial when a sentence stops you cold: "The agent updates its Q-value using the discounted return from the trajectory." You know…
Read tutorial
The reward curve climbs steadily. The training loss falls exactly as expected. And the outputs get visibly worse. Human reviewers start rejecting what the…
Read tutorial
Two policies can share the exact same expected return and still feel completely different to the person who has to live with the outcomes. One delivers…
Read tutorial
A planner is only as stable as the model it trusts. Change one transition probability by a few points and watch the preferred action move.
Read tutorial
A higher estimated value is not proof of improvement. It is a hypothesis about actions the dataset may never have supported.
Read tutorial
Two agents start on the same grid. Same states, same actions, same rewards, same exploration schedule. One hugs the edge of a cliff and occasionally falls.…
Read tutorial
A self-play win rate can climb for a thousand iterations and still be lying to you. Here is the smallest instrument that catches it.
Read tutorial
Most actor-critic methods force you to choose between sample efficiency and stable exploration. Soft Actor-Critic (SAC) refuses the trade—and that refusal…
Read tutorial
You add an entropy bonus to your objective, and the whole value structure shifts under your feet. The max in the Bellman backup becomes a log-sum-exp. The…
Read tutorial
A SAC actor hands you two tensors: the action you just sent to the environment, and logp_pi. That second number is not the Gaussian log-density of the…
Read tutorial
A Q-table makes the invisible visible. Before neural networks, before GPUs, before thousands of parameters, there is a simple grid, a handful of numbers,…
Read tutorial
DDPG doesn't fail because its policy is weak. It fails because its critic quietly starts lying, and the policy trusts every word.
Read tutorial
You have trained a policy that works. The weights are tuned, the rewards are climbing, and the agent is finally doing something that looks like competence.…
Read tutorial
The real question is not which family of algorithms is better. It is which one fits the structure of your problem—your action space, your data budget, and…
Read tutorial
You can recite that advantage equals Q minus V and still have no way to tell whether a program that prints those numbers is correct. That gap between…
Read tutorial
You're reading an actor-critic explanation and you hit three symbols: V, Q, and A. The critic estimates V. The policy gradient weights updates by A.…
Read tutorial
A flat agent that learns the easy part of a long task and then stalls is not asking for a hierarchy. It is asking for a diagnosis.
Read tutorial