Direct policy optimization using gradients of expected outcomes, including estimator construction, baselines, and actor-critic updates.
Tagged articles
19 articles in this tag.
If you've worked through policy gradients, you've felt the problem: one lucky episode can make a mediocre action look brilliant, and the next update swings…
Read tutorial
A team wins. The shared reward is +1. Every agent's gradient gets the same number, so the agent who stood still learns exactly as much as the agent who…
Read tutorial
You can write PPO's loss in three lines of code and still not know why the min() is there. That gap matters, because the min() is the whole algorithm.
Read tutorial
The deterministic actor loss is one line: maximize $Q(s, \mu_\theta(s))$. It looks like a trick because the log-probability term — the entire engine of the…
Read tutorial
A stochastic policy averages over actions. A deterministic policy climbs the critic's slope at the single action it outputs. That swap—from averaging to…
Read tutorial
A config file hands you lambda=0.95. You change it to 0.99, the run gets worse, you change it back, and you still cannot say what the number means. That is…
Read tutorial
Every actor-critic user hits the same wall shortly after their first working implementation: the advantage estimate looks fine in theory, but the policy…
Read tutorial
You can read PPO's clipped objective ten times and still not know whether your implementation works. The curve tells you. So does the clip fraction. So…
Read tutorial
You have read the claim: subtract an action-independent baseline and the policy gradient keeps the same expected value while its variance drops. It is easy…
Read tutorial
Two runs of the same policy-gradient code. Same environment, same seed, same learning rate. One climbs. One flatlines. The usual advice is "subtract a…
Read tutorial
That gap between a falling loss and a falling pole is the first thing to fix, because it tells you which number you are actually watching. In a…
Read tutorial
You've built a Q-learning agent. It works. For each state, you compare action values, pick the best one, and call that a policy. Then someone mentions…
Read tutorial
The REINFORCE update looks like a magic trick: multiply a log-probability by a return, add it to the parameters, and hope the policy improves. It is not a…
Read tutorial
A policy-gradient update is a stride, not a leap. Take too long a step, and you can land somewhere worse than where you started—sometimes so much worse…
Read tutorial
Most actor-critic methods force you to choose between sample efficiency and stable exploration. Soft Actor-Critic (SAC) refuses the trade—and that refusal…
Read tutorial
DDPG doesn't fail because its policy is weak. It fails because its critic quietly starts lying, and the policy trusts every word.
Read tutorial
You bump clip_range from 0.2 to 0.3. The curve climbs higher. You conclude clipping matters.
Read tutorial
You have trained a policy that works. The weights are tuned, the rewards are climbing, and the agent is finally doing something that looks like competence.…
Read tutorial
The real question is not which family of algorithms is better. It is which one fits the structure of your problem—your action space, your data budget, and…
Read tutorial