Push policy learning further with stable updates, entropy, advantage estimation, continuous actions, TRPO, SAC, TD3, and modern actor-critic methods.
Tutorials
Follow the learning path in order or jump directly to the RL concept or experiment you need.
You have trained a policy that works. The weights are tuned, the rewards are climbing, and the agent is finally doing something that looks like competence.…
Read tutorial
Most reinforcement learning tutorials treat randomness as a ladder: you need it to explore, and once training converges, you kick it away. Maximum-entropy…
Read tutorial
Most actor-critic methods force you to choose between sample efficiency and stable exploration. Soft Actor-Critic (SAC) refuses the trade—and that refusal…
Read tutorial
You add an entropy bonus to your objective, and the whole value structure shifts under your feet. The max in the Bellman backup becomes a log-sum-exp. The…
Read tutorial
Your SAC implementation is probably correct. That is exactly why the failed run is so frustrating: the losses move, the entropy term behaves, and the agent…
Read tutorial
A stochastic policy averages over actions. A deterministic policy climbs the critic's slope at the single action it outputs. That swap—from averaging to…
Read tutorial
Every actor-critic user hits the same wall shortly after their first working implementation: the advantage estimate looks fine in theory, but the policy…
Read tutorial
A config file hands you lambda=0.95. You change it to 0.99, the run gets worse, you change it back, and you still cannot say what the number means. That is…
Read tutorial
A GAE implementation can be wrong in a way that still prints plausible numbers. The terminal mask is where that happens.
Read tutorial
DDPG doesn't fail because its policy is weak. It fails because its critic quietly starts lying, and the policy trusts every word.
Read tutorial
You move from discrete actions to continuous control, wire up a Gaussian policy head, and the agent either jitters uselessly around a target or slams…
Read tutorial
The deterministic actor loss is one line: maximize $Q(s, \mu_\theta(s))$. It looks like a trick because the log-probability term — the entire engine of the…
Read tutorial
A SAC actor hands you two tensors: the action you just sent to the environment, and logp_pi. That second number is not the Gaussian log-density of the…
Read tutorial