beginner
13 min read
How to Design a Reinforcement Learning Environment
Most beginners assume the algorithm is the hard part of reinforcement learning. Pick the right one, tune it well, and the agent will learn. Then they spend…
Read tutorialintermediate
Designing and Evaluating RL Experiments
Find out whether an agent is actually learning: design environments, choose metrics, diagnose failure, compare baselines, and test generalization.
Tutorials
Articles in Designing and Evaluating RL Experiments
Follow the learning path in order or jump directly to the RL concept or experiment you need.
intermediate
8 min read
How to Evaluate Reinforcement Learning Agents Beyond One Good Episode
Your agent just nailed an episode. It reached the goal, collected a beautiful stream of rewards, and you felt that rush of validation. Then you rerun it,…
Read tutorial
intermediate
10 min read
Why Reinforcement Learning Agents Fail: A Diagnostic Guide
Your agent runs. The loop completes. No exceptions are raised. And yet the reward curve sits there like a flatline—or climbs promisingly for two hundred…
Read tutorial
intermediate
11 min read
Reward Shaping vs Sparse Rewards in Reinforcement Learning
Your agent is doing one of two things: nothing, or the wrong thing. When rewards never arrive, it wanders forever without learning. When rewards arrive too…
Read tutorial
advanced
9 min read
Why Potential-Based Reward Shaping Preserves Policy Rankings
A shaped agent learns faster, the curve looks great, and the final policy is subtly wrong. The shaping term did not speed up learning; it quietly rewrote…
Read tutorial
intermediate
11 min read
Test Potential-Based Reward Shaping in a Small RL Task
"Shaping speeds up learning without changing the optimal policy." That claim is true, and it is also the most misread sentence in reinforcement learning.…
Read tutorial
intermediate
10 min read
Baselines and Ablations for Reinforcement Learning Experiments
You changed two things at once. The agent got better. And now you have no idea which change earned the improvement.
Read tutorial
absolute beginner
9 min read
A Reinforcement Learning Learning Path: From Agent Loop to Deep RL
Most people start reinforcement learning the wrong way. They install a deep RL library, copy a DQN or PPO example, watch the agent fail—or worse, behave…
Read tutorial
beginner
9 min read
Learning Rate and Discount Factor in RL: A Small Experiment With Different Behaviors
Two knobs. One update rule. If you are starting out in reinforcement learning, it is tempting to treat the learning rate and discount factor as twin dials…
Read tutorial
intermediate
9 min read
The Deadly Triad in Reinforcement Learning: Why Learning Can Become Unstable
Your Q-learning agent trains beautifully for ten thousand steps. The loss curve dips, the reward trend climbs, and you start to believe the hard part is…
Read tutorial
intermediate
12 min read
Generalization in Reinforcement Learning: Does the Policy Transfer?
Your agent crushes its training level. Every episode, the same flawless path, the same high score, the same satisfying convergence curve. Then you move one…
Read tutorial
intermediate
9 min read
Invalid Actions in Reinforcement Learning: Masks, Penalties, and Hidden Bugs
Your agent keeps walking into walls. It selects skills still on cooldown, tries to buy items it cannot afford, and bumps against the grid edge like a moth…
Read tutorial
advanced
11 min read
Offline Reinforcement Learning Explained: Learning From a Fixed Dataset
An offline RL agent can report a sky-high value for an action it never actually tried. The critic is extrapolating into territory the dataset never…
Read tutorial
advanced
10 min read
Extrapolation Error in Offline RL: Why Unseen Actions Mislead the Agent
Imagine training an offline agent and watching its Q-values climb steadily. The critic grows more confident. The policy selects actions with increasing…
Read tutorial
advanced
12 min read
Sim-to-Real Reinforcement Learning: Why a Policy Breaks Outside the Simulator
A policy that scores perfectly in simulation is not a policy that works in the real world. It is a policy that solved the simulator.
Read tutorial
intermediate
13 min read
Off-Policy Evaluation in Reinforcement Learning: Estimating a Policy From Logged Experience
You have a new policy that looks promising. The next step feels obvious: deploy it, watch it interact with the real environment, and measure what happens.…
Read tutorial
advanced
11 min read
Safe Policy Improvement From Logged Data: Staying Near What the Dataset Supports
A higher estimated value is not proof of improvement. It is a hypothesis about actions the dataset may never have supported.
Read tutorial
intermediate
11 min read
Dataset Support and Coverage in Offline RL: Can the Logs Answer Your Question?
You train an offline agent on a fixed dataset. The final score looks strong. The learning curves climb. You feel the quiet satisfaction of a result that…
Read tutorial
advanced
11 min read
Importance Sampling and Doubly Robust Evaluation in Offline RL
You have a dataset of logged trajectories collected under one policy, and you want to know what a different policy would have earned. The naive…
Read tutorial
advanced
8 min read
Constrained Reinforcement Learning: Optimizing Reward Without Breaking the Rules
A penalty is a price. A constraint is a wall. Most RL safety problems fail because practitioners build walls out of prices and then act surprised when the…
Read tutorial
advanced
11 min read
Safe Exploration in Reinforcement Learning: Action Shields, Rules, and Fallbacks
Your agent proposes an action that crosses a hard safety boundary. You can block it, replace it, or fall back to a known-safe controller. The naive…
Read tutorial
advanced
11 min read
When Can an Action Shield Support a Safety Claim?
A shield that rejects unsafe actions every run can still support no safety claim at all.
Read tutorial
intermediate
10 min read
Add and Test an Action Shield During Reinforcement Learning Exploration
Your agent keeps taking an action you already know is wrong. You add a shield. Now you cannot tell whether it is helping or quietly starving the learner.
Read tutorial
advanced
14 min read
Risk-Sensitive Reinforcement Learning: Choosing Policies Beyond Average Return
Two policies can share the exact same expected return and still feel completely different to the person who has to live with the outcomes. One delivers…
Read tutorial
intermediate
11 min read
CVaR Objectives for Risk-Sensitive Reinforcement Learning
Two policies. Same expected return. One of them occasionally produces an episode so bad you would never deploy it. The mean cannot tell them apart — not…
Read tutorial
intermediate
8 min read
Compare Mean Return and Downside-Sensitive Policy Choices in a Toy Task
Two policies. One wins on average. The other wins on the days you actually have to survive. If your selection rule can't tell them apart, you're not…
Read tutorial
advanced
11 min read
Evaluating Exploration in RL: Did the Agent Discover Useful Information?
A novelty bonus can make your agent look busy without making it look smart. The intrinsic-reward curve climbs, the state-visitation heatmap fills in, and…
Read tutorial
intermediate
16 min read
Compare Offline Policy-Value Estimates on Logged Trajectories
Two estimators, one dataset, two different answers. That gap is not a bug in the math — it is the most useful signal you will get about whether your logs…
Read tutorial
intermediate
10 min read
Compare Penalty and Primal-Dual Methods in a Small Constrained MDP
A hand-picked penalty weight can look fine in one run and drift over budget in the next. The weight was never told what the budget is — so it can't respect…
Read tutorial
intermediate
10 min read
Test Reinforcement Learning Policy Generalization Across Held-Out Environments
Your policy scores well on the training environment. You change one parameter, run it again, and the score falls off a cliff. Is that a transfer failure,…
Read tutorial
advanced
10 min read
Derive the Lagrangian and Primal-Dual Update for a Constrained MDP
You write down the problem cleanly: maximize expected reward, keep expected cost under a budget. Then you freeze, because there is no obvious place to put…
Read tutorial
intermediate
10 min read
Diagnose Simulator-to-Deployment Mismatches From Paired Traces
Your policy scored well in simulation. On the target system, it behaves differently. All you have is two logs of the same episode — one from each system —…
Read tutorial
intermediate
8 min read
Derive a Conservative Policy-Improvement Rule From Logged Data
A candidate policy that looks better on paper is not the same as a candidate policy that is better. Here is one small, honest rule for telling the…
Read tutorial
intermediate
8 min read
Trace a Linear Counterexample to the Deadly Triad
Everyone repeats that the deadly triad diverges. Almost nobody watches it happen. So let's build a predictor small enough to update by hand, then watch the…
Read tutorial
advanced
10 min read
Derive Importance Sampling and Doubly Robust Off-Policy Evaluation
You have a log of episodes from one policy, a candidate policy you have never deployed, and a single number that claims to be the candidate's value. The…
Read tutorial
intermediate
9 min read
Uncertainty in Reinforcement Learning Evaluation: Comparing Runs and Policies
Two algorithms. Five seeds each. A forty-point gap in mean return. Someone asks which one to ship, and the honest answer is: you do not know yet. Not…
Read tutorial
intermediate
9 min read
Generalization in Reinforcement Learning: Task Distributions and the Generalization Gap
A policy scores well on a handful of held-out levels. The write-up says it "generalizes to unseen environments." Then deployment happens, and the policy…
Read tutorial