Transition Dynamics and Reward Models in Reinforcement Learning

You can name the parts of an MDP. States, actions, rewards, transitions—you have the vocabulary down. But if I ask you what the environment model actually predicts, the answer probably gets fuzzy.

That gap is normal. Most introductions hand you the MDP as a formal tuple and move on. What gets lost is the difference between a single observed step and the model that describes every step the environment could produce. Confuse the two, and you will misread what your agent knows, why it plans badly, and where its failures actually come from.

Let's make the split concrete.

One step is not the whole story

Picture a small robot pushing a block across a slippery floor. The block sits at position A. The robot applies a push to the right.

What happens next?

If you have watched only one push, you might say the block lands at position B. That was what happened last time. But floors vary. Sometimes the block slides further. Sometimes it stops early. Sometimes it drifts off course. The same action, from the same state, can produce several different outcomes.

That single observed push—one state, one action, one next state, one reward—is a sample. It is one draw from a larger set of possibilities. The environment model is the description of that entire set: every place the block could land, and how likely each landing is.

The environment model in reinforcement learning predicts two things:

1. Where you land after taking an action from a state.
2. What you earn when you get there.

Everything else in RL builds on those two predictions. Value functions estimate long-term consequences using them. Policies decide actions based on what those consequences are worth. But before you can reason about value, you need a clear picture of what the environment itself is telling you.

What the environment model actually contains

Before we go further, let's pin down the names, because the literature throws several labels at the same ideas.

The environment model is the umbrella term. It usually has two parts:

• The transition model, often written P, which describes how the environment changes. "Transition dynamics" is another name for the behavior this model captures.
• The reward model, often written R, which describes the feedback signal the environment returns.

A single observed step—one transition and one reward—is not either model. It is one piece of evidence about them.

Transition dynamics: the distribution over where you land

The transition model answers a question with probabilities: If I take action a from state s, what states can I end up in, and with what likelihood?

Formally, it is written as P(s′ | s, a)—the probability of landing in next state s′ given that you started in state s and took action a. But the notation hides the important part. This is not a single outcome. It is a probability distribution over all possible next states.

Let's return to the robot. From state A with a rightward push, the possible outcomes might look like this:

The transition dynamics are the whole table. Not the row that happened to occur. The table.

This is what makes an environment stochastic: the same action does not always produce the same result. Wind, friction, noise, dice rolls—anything that introduces variability into the environment creates a spread of possible next states.

Some environments are deterministic. In those cases, the table collapses: one next state has probability 1, and all others have probability 0. Push the button, the light turns on. Every time. Deterministic environments are a special case of the distribution view, not a different kind of system. The model still describes possibilities; it just happens that only one possibility exists.

Here is the mental model worth keeping: the state transition model is the full distribution. Any single step you observe is just one sample drawn from it. If you watch the robot push the block once and it lands at B, you have learned one fact about the environment. You have not learned the distribution. To estimate the distribution, you need many pushes, or some other way to infer the underlying pattern.

Reward models: the signal that grades the landing

The reward model answers a related question: When the environment moves you from one state to another, what scalar signal does it return?

That signal is the agent's only direct feedback about whether its behavior is good or bad. The reward model is the rule that produces that feedback.

Reward models come in a few common forms, depending on what the reward depends on:

• R(s) — reward for simply being in a state.
• R(s, a) — reward for taking a specific action from a state.
• R(s, a, s′) — reward for the full transition: starting in s, taking a, and landing in s′. This is the most general form, because the reward can depend on where you actually ended up.

That last form matters for stochastic environments. If the robot's push can land the block in several places, each landing might carry a different reward. Push right and land at B: reward +1. Push right and slide too far to C: reward −2, because the block overshot the target zone.

One boundary is worth stating clearly. In many formulations—including our robot example—the reward is a fixed value attached to the realized transition (s, a, s′). Once you know where you landed, you know what you earned. In other formulations, the environment samples the reward from its own distribution. The transition model is always a distribution. The reward model is often a simple mapping, but it can also be a conditional distribution when the environment itself is noisy about payoffs. For now, treat rewards as attached to the landing: the transition determines what you earn.

Model predictions vs. observed transitions: why the split matters

Here is the distinction that separates a clear understanding of RL from a muddled one:

• The model is a compressed description of how the environment behaves. It says: from this state, with this action, here are the possible next states, their probabilities, and the rewards attached to them.
• A transition sample is one observed fact: from this state, with this action, the environment moved me here and gave me this reward.

One is the map. The other is a single journey across the terrain.

Why does this distinction matter? Because of what an agent can do with each.

An agent that has seen one outcome of an action does not know the distribution. It knows one data point. If it treats that data point as the whole truth, it will act as though the environment is deterministic when it is not. The robot pushes the block once, it lands at B, and the agent concludes pushes always land at B. Then the floor gets slippery, the block slides to C, and the agent's plan falls apart—not because the plan was bad, but because the model underneath it was wrong.

This is the failure mode to internalize: treating one lucky or unlucky sample as the true dynamics leads to wrong predictions and bad planning.

A model can be given to the agent in advance or estimated from the samples it collects. When an agent plans, it uses an explicit predictive model to imagine outcomes before acting. Model-free methods take a different route: they learn values or policies directly from experience without maintaining that explicit predictive model. Both approaches are valid; they make different tradeoffs. But you cannot understand the tradeoff until you see that the model and the samples are different things.

Why the split matters for planning and diagnosis

Planning requires the model. When an agent plans, it considers possible futures before acting: If I push right, I might land at B, C, or D. B is worth more. I'll push right. That reasoning requires the distribution, not just past samples. A single past push tells the agent what happened once. Planning needs to know what could happen next, and how likely each possibility is.

This is also where diagnosis gets interesting. When an agent behaves oddly, the cause is often hiding in one of the two model components:

• Wrong transition model: the agent expects to land somewhere it rarely lands. It plans for a world that does not exist.
• Wrong reward model: the agent lands exactly where it expects, but earns something different than it predicted. The world behaves as expected; the agent just misjudged what it was worth.

I have seen this confusion sink more than one debugging session. Someone watches an agent fail and assumes the policy is broken. But the policy is just the visible layer. Underneath it sit the agent's beliefs about where actions lead and what those landings pay. If either belief is wrong, the policy will look irrational even when it is perfectly rational given what the agent believes.

So when an agent misbehaves, start with two questions in order:

1. Did it mispredict where it would land?
2. Did it mispredict what it would earn?

These are the first two checks, not an exhaustive list—policy errors, representation problems, and poor value estimates can also cause trouble. But transition and reward beliefs are where I would look first, because they are the foundation everything else stands on.

The durable mental model

Keep this split close:

• The environment model describes where you can land and what you can earn.
• An observed step is one draw from that description.

The model is the whole table of possibilities. The sample is a single row that happened to occur. Planning requires the table. Diagnosis requires checking whether the table is accurate.

When you move on to value functions and Bellman reasoning, this foundation will carry you. Value functions are built on top of transition and reward models—they ask, given these dynamics, how much future reward can I expect from here? If the dynamics are fuzzy in your head, the value equations will feel like algebra without meaning. If the dynamics are clear, the equations become what they actually are: a way to summarize long-term consequences from short-term predictions.

So before you chase algorithms, make sure you can answer one question about any environment you meet: What does the model predict, and what did the agent actually observe? The gap between those two answers is where most of reinforcement learning happens.