beginner
14 min read
Exploration vs Exploitation in Reinforcement Learning
Your agent is stuck in a loop. It found one action that earns a small reward, and now it repeats that action forever—never discovering that a much better…
Read tutorialbeginner
Learning From Experience: Exploration and Temporal Difference
See how agents learn by trying: explore, collect experience, assign credit across time, and improve when the environment is unknown.
Tutorials
Articles in Learning From Experience: Exploration and Temporal Difference
Follow the learning path in order or jump directly to the RL concept or experiment you need.
intermediate
10 min read
Monte Carlo vs Temporal-Difference Learning in Reinforcement Learning
Both methods learn from experience. One waits for the story to end before drawing conclusions. The other revises its belief after every sentence. The real…
Read tutorial
intermediate
11 min read
Q-Learning Explained: How Off-Policy Value Learning Finds Better Actions
You can recite the Q-learning update. Maybe you've even coded it in a toy grid world and watched the agent stumble toward a goal. But if someone asked you…
Read tutorial
beginner
9 min read
Deriving the Q-Learning Update: Targets, Learning Rates, and a Worked Example
You can copy the Q-learning update into code and watch it work. Then someone asks why the target uses the best next action when the agent just took a…
Read tutorial
beginner
10 min read
Implement Tabular Q-Learning and Inspect the Q-Table
The Q-learning update looks like a ritual until you print the table before and after training and watch the numbers crawl toward the goal.
Read tutorial
intermediate
8 min read
SARSA vs Q-Learning: On-Policy and Off-Policy Learning Compared
Two agents start on the same grid. Same states, same actions, same rewards, same exploration schedule. One hugs the edge of a cliff and occasionally falls.…
Read tutorial
intermediate
7 min read
Credit Assignment in Reinforcement Learning: Connecting Late Rewards to Earlier Actions
The agent makes a move. Nothing happens. It makes another move, then another. Twenty steps later, a reward arrives—and the agent has no idea which action…
Read tutorial
beginner
9 min read
A Small Tabular RL Experiment: Watch Exploration Become a Policy
A Q-table makes the invisible visible. Before neural networks, before GPUs, before thousands of parameters, there is a simple grid, a handful of numbers,…
Read tutorial
intermediate
11 min read
Eligibility Traces in Reinforcement Learning: A Bridge Across Time
One-step TD updates are beautifully local: take a step, observe the reward and next state, correct the value of the state you just left. Then you hit the…
Read tutorial
intermediate
11 min read
Forward and Backward Views of Eligibility Traces: Deriving TD(λ)
You read one chapter that builds TD(λ) from a weighted mixture of n-step returns. You read another that updates a decaying trace and broadcasts the TD…
Read tutorial
intermediate
8 min read
Experiment With TD(λ) and Eligibility Traces in a Small RL Task
Watch a one-step TD learner on a short corridor. The reward sits at the far end, and the value of the first state crawls upward for hundreds of episodes.…
Read tutorial
intermediate
8 min read
Q-Learning Overestimation Bias: Why the Max Can Fool the Agent
Your agent's Q-values are climbing. Its actual performance is flatlining. And the gap between what the agent believes and what it achieves keeps widening.
Read tutorial
beginner
9 min read
Contextual Bandits vs Reinforcement Learning: When Actions Don’t Shape Future States
Picture a system that picks which article to show a returning reader. It looks at what it knows about them—their reading history, the time of day, the…
Read tutorial
advanced
11 min read
Intrinsic Motivation in Reinforcement Learning: Exploring When External Rewards Are Sparse
Curiosity is a loan of learning signal. It accelerates exploration when the borrowed interest points toward the goal—and it becomes a debt when the agent…
Read tutorial
intermediate
10 min read
Behavior Cloning and Covariate Shift: Why Demonstrations Fail Off the Training Path
A behavior-cloned agent can look flawless on training data and fall apart the moment it makes one small mistake. That is not bad luck. It is the signature…
Read tutorial
intermediate
13 min read
Derive How Behavior-Cloning Errors Grow Across a Trajectory
A cloned policy can score 98% on held-out demonstration states and still walk off a cliff in four seconds. The metric and the rollout are measuring two…
Read tutorial
beginner
10 min read
Experiment With Behavior Cloning and Covariate Shift in Python
A cloned policy can score 100% on the states it trained on and still walk off the path the moment it drives itself. That gap is the whole experiment.
Read tutorial
beginner
12 min read
Imitation Learning vs Reinforcement Learning: Copying Behavior or Optimizing Outcomes?
You have a stack of expert demonstrations. Maybe you recorded a human driving a car, or you logged the decisions of a system that already works well. The…
Read tutorial
advanced
10 min read
Inverse Reinforcement Learning: Inferring Goals From Demonstrations
Demonstrations tell you what an expert did. Inverse reinforcement learning asks why—and the answer is never uniquely determined by the behavior itself.
Read tutorial
advanced
11 min read
Exploration Bonuses in RL: Count, Novelty, Prediction Error, and Uncertainty
Epsilon-greedy exploration is randomness without a reason. An exploration bonus is randomness with a budget and a purpose: it changes the reward signal…
Read tutorial
advanced
10 min read
Combining Demonstrations With Reinforcement Learning
Pure reinforcement learning wastes samples rediscovering what a demonstrator already solved. Pure imitation freezes at the demonstrator's ceiling. The real…
Read tutorial
beginner
8 min read
Build and Verify a Tiny Tabular RL Environment in Python
A learning algorithm that never improves is usually blamed for the wrong crime. More often, the environment quietly hands back the wrong next state, pays…
Read tutorial
beginner
9 min read
Evaluate a Tabular RL Agent Across Repeated Runs
A single episode is one sample. A single seed is one opinion. If you want to know whether your tabular agent learned anything, stop reading one number and…
Read tutorial
beginner
9 min read
Double Q-Learning Explained: Separate Action Selection From Evaluation
Your Q-table has a favorite action. It picked that action because its estimate was highest, and now it is about to use that same estimate to decide how…
Read tutorial
intermediate
11 min read
Optimism and Confidence Bonuses in Tabular Reinforcement Learning
An agent that has seen an action once does not know its value. It knows one sample. Optimism is the discipline of treating thin evidence as a reason to…
Read tutorial
intermediate
12 min read
When Does Tabular Q-Learning Converge? Conditions and Limits
Your Q-table went quiet. The values stopped moving, the greedy policy stopped changing, and the learning curve flattened into something that looks like a…
Read tutorial
beginner
9 min read
Compare Greedy and Exploratory Strategies in a Contextual Bandit With Python
A greedy agent looks brilliant for the first fifty rounds. Then it stops learning, because it stopped asking questions.
Read tutorial
intermediate
7 min read
Compare Count and Uncertainty Bonuses in a Small Reinforcement Learning Task
You add a curiosity bonus. Coverage climbs. The return curve goes flat. That gap is the whole lesson: a bonus is a second reward channel, and the only…
Read tutorial
intermediate
10 min read
Test How Demonstrations Shape Later Online Reinforcement Learning
A policy cloned from demonstrations can look brilliant and still be useless. It scores well on the states it was shown, then stalls the moment the world…
Read tutorial
intermediate
9 min read
Explore Reward Ambiguity With a Tiny Inverse Reinforcement Learning Example
Two reward functions. One demonstrated path. No way to tell them apart from behavior alone.
Read tutorial
beginner
8 min read
Measure Q-Learning Overestimation With a Noisy-Value Experiment
You have read that Q-learning overestimates. You nodded. You moved on. But you have never seen the bias as a number, so it stays a rumor.
Read tutorial
beginner
11 min read
Compare SARSA and Q-Learning on a Risky Path
Two agents train on the same grid, with the same rewards, the same learning rate, and the same epsilon. One learns to hug the edge of a penalty band. The…
Read tutorial
intermediate
8 min read
Test When Curiosity Rewards Distract an RL Agent
Watch the two curves long enough and you will see the moment curiosity stops working for the agent and starts working for itself. The intrinsic bonus keeps…
Read tutorial
intermediate
13 min read
Derive Cumulative Regret in a Finite Contextual Bandit
A policy that earns more reward is not automatically a better policy. It may just have met easier contexts.
Read tutorial
intermediate
13 min read
Deriving N-Step TD Returns: Bootstrapping Between One-Step TD and Monte Carlo
One-step TD updates fast but bets on a guess. Monte Carlo waits for the whole episode but trusts only what happened. The n-step return asks a sharper…
Read tutorial
intermediate
9 min read
Deriving the Expected SARSA Target With a Worked Action-Value Example
You know the SARSA target. You know the Q-learning target. Then someone says "Expected SARSA" and your brain stalls on one word: expectation. What action…
Read tutorial