Skip to content

beginner

Value Functions and Bellman Reasoning

Turn experience into predictions about future outcomes, then connect value, action value, advantage, and Bellman relationships to observable decisions.

Tutorials

Articles in Value Functions and Bellman Reasoning

Follow the learning path in order or jump directly to the RL concept or experiment you need.

An IT professional operates a computer in a server room, managing network systems and connected devices.

beginner

10 min read

Value Functions in Reinforcement Learning: Predicting Future Reward

A reward tells you what just happened. A value function predicts the return you can expect from here to the end of the episode—and that forward-looking…

Two playing cards and poker chips, including a Las Vegas chip, on a grey background. Ideal for casino or gambling themes.

beginner

12 min read

The Bellman Equation Explained Through a Decision Tree

The first time you see the Bellman equation, it looks like a symbol puzzle. Greek letters, nested expectations, a value function that seems to appear on…

From below of monitor of modern computer with opened files on blue screen

intermediate

11 min read

Value vs Q-Value vs Advantage in Reinforcement Learning

You're reading an actor-critic explanation and you hit three symbols: V, Q, and A. The critic estimates V. The policy gradient weights updates by A.…

A futuristic robot dog, the Cyberdog, on display in an indoor setting, showcasing advanced robotics technology.

intermediate

10 min read

Model-Based vs Model-Free Reinforcement Learning: What Does the Agent Know?

The real difference between model-based and model-free reinforcement learning is not how much the agent learns. It is whether the agent can predict what…

A young boy engages with a humanoid robot during an indoor tech exhibition, symbolizing future innovation.

intermediate

12 min read

Markov Decision Processes: The Formal Map Behind Reinforcement Learning

The agent-environment loop feels simple until you try to make an algorithm learn. Then the loop stops being enough. The agent needs to know what it can…

Close-up of wooden chess pieces on a board, emphasizing strategy and tactics.

intermediate

12 min read

Bellman Backups and Dynamic Programming for Reinforcement Learning

A Bellman equation states a consistency condition. A Bellman backup is the mechanical step that enforces it. The difference between reading an equation and…

Peaceful beach scene with an upside down boat and straw huts under a blue sky.

beginner

13 min read

Prediction vs Control in Reinforcement Learning: Evaluating Policies and Improving Them

Scoring a policy and improving a policy feel like the same job. They are not. One is a measurement. The other is a search. Confusing them is one of the…

A man in a plaid shirt reaches for poker chips on a white surface, depicting a gambling scene.

intermediate

9 min read

Planning With a Learned Model in Reinforcement Learning

An agent that only learns from real experience is sample-hungry. An agent that trusts its own imagined experience can be confidently wrong. Planning with a…

A blue Yaskawa industrial robot arm on display, showcasing advanced technology and robotics.

beginner

9 min read

Transition Dynamics and Reward Models in Reinforcement Learning

You can name the parts of an MDP. States, actions, rewards, transitions—you have the vocabulary down. But if I ask you what the environment model actually…

3D rendered abstract brain concept with neural network.

intermediate

11 min read

Why Bellman Updates Converge: A Fixed-Point Mental Model for RL

Value iteration is not a loop that guesses better. It is a machine that shrinks the distance between any two value estimates by a factor of gamma on every…

A robotic hand reaching into a digital network on a blue background, symbolizing AI technology.

intermediate

12 min read

Policy Improvement and Generalized Policy Iteration in Reinforcement Learning

You can compute value estimates. You can trace a Bellman backup through a small grid. But if someone asked you to explain how those numbers actually change…

View of the Skylon Tower and casino against a clear blue sky in Niagara Falls, Ontario.

intermediate

9 min read

Compounding Model Error in Reinforcement Learning: Why Planning Drifts

A model that predicts the next state almost perfectly can still produce plans that drift into nonsense after a few imagined steps. The mistake is treating…

Vintage cars parked outside a neon-lit casino at night, showcasing urban nightlife.

advanced

9 min read

Distributional Reinforcement Learning Explained: Predicting Return Distributions Instead of Averages

Two actions can offer the same expected return and still feel completely different to the agent that takes them. One delivers a steady, predictable…

A detailed close-up of chess pieces on a board, highlighting the knight in warm tones.

advanced

13 min read

Dyna-Style Reinforcement Learning: Mixing Real and Imagined Experience

Every real transition in a Dyna-style agent does two jobs at once: it teaches the value learner and it teaches the model. Then the model becomes a second…

Close-up of a yellow Ethernet cable with connectors on a blue background.

advanced

8 min read

Belief States in POMDPs: Acting on Probabilities About Hidden State

A policy that acts on the latest observation alone is guessing. A belief state is how you stop guessing.

Close view of playing cards and hand on a casino table, capturing the gambling atmosphere.

advanced

11 min read

Options in Reinforcement Learning: How Temporally Extended Actions Work

A flat policy re-decides everything, every step. Options let it decide once and commit.

A captivating black and white photo of a chess game with fallen king piece, symbolizing defeat.

advanced

13 min read

When Hierarchical Reinforcement Learning Helps: Subgoals, Skills, and Horizon

A flat agent that learns the easy part of a long task and then stalls is not asking for a hierarchy. It is asking for a diagnosis.

A stunning aerial view of Macau's skyline under a bright blue sky, featuring iconic casino architecture.

advanced

7 min read

Average-Reward Reinforcement Learning: Learning in Continuing Tasks

Your agent has no finish line. No terminal state ends the run, no episode boundary resets the world, and no natural horizon cuts the future off. You reach…

Glad young female chess player with small piece leaning on hand while sitting near board in park and looking at camera

advanced

11 min read

Model Predictive Control for Reinforcement Learning: Plan, Act, Replan

A ten-step plan is a promise your model cannot keep. Execute step one, measure what actually happened, and throw the other nine away.

A close-up of poker chips and playing cards on a casino gaming table, highlighting Ace of Diamonds.

advanced

10 min read

Choosing a Planning Horizon in Model-Based RL

The agent plans beautifully for five steps. Then the imagined future quietly stops matching reality, and the only knob you reach for is horizon length.