Reinforcement Learning Terms: A Practical Reference for States, Rewards, Values, and Episodes

You're halfway through a tutorial when a sentence stops you cold: "The agent updates its Q-value using the discounted return from the trajectory." You know what an agent is. You have a vague sense about rewards. But Q-values? Discounts? Trajectories? The sentence might as well be in another language.

Here's the thing: reinforcement learning terms feel overwhelming because they arrive out of order. Tutorials toss them at you mid-explanation, each one floating without a place on the decision loop it belongs to. An alphabetical glossary doesn't fix that—it just hides the relationships.

This guide takes a different approach. Instead of listing terms alphabetically, we'll group them by the job they do in the learning process. Think of it as a map rather than a dictionary. When you hit an unfamiliar term in your next tutorial, you'll know which part of the map it belongs to—and that's half the battle.

The Decision Loop: Agent, Environment, State, Action, Reward

Every reinforcement learning idea hangs on one loop. Master this loop and the rest of the vocabulary starts to feel like variations on a theme.

Picture a robot learning to navigate a simple grid. The robot is the agent—the learner and decision-maker. Everything outside the robot—the grid, the walls, the goal square—is the environment. The environment doesn't learn. It just responds.

At each moment, the robot receives a state: the information that fully describes its situation for the task at hand. In the grid example, the state might be the robot's current position. In a video game, the state might be the current screen.

The robot then picks an action—a choice from its available options. Move left. Move right. Move up. The full set of available choices is the action space. In chess, the action space is every legal move. In a game with a joystick, it might be every direction the stick can tilt.

Here's where the loop gets interesting. The environment responds to the action in two ways. First, it moves to a new state. Second, it sends back a reward—a single number that says how good that outcome was. The reward might be +1 for reaching the goal, −1 for hitting a wall, and 0 for every other step.

Trace one full cycle: the robot observes its position, chooses to move right, receives a reward of 0, and arrives at a new position. Then it repeats. Observe. Choose. Receive. Move. That cycle is the heartbeat of reinforcement learning.

Note: The reward is feedback about the last action, not a grade on the agent's overall performance. A single reward tells the agent how that one step went. Learning what leads to good rewards over time is the harder problem—and that's where the rest of these terms come in.

States and Observations: What the Agent Actually Sees

If you read further into RL, you'll meet a distinction that looks subtle at first but changes how you interpret entire algorithms: the difference between a state and an observation.

A state is the information needed to describe the situation for the task. In a fully observed problem, the agent gets the complete state. The robot on the grid knows its exact position, so what it receives is the state.

An observation is what the agent actually receives. In many problems, the observation and the state are the same thing. But in a partially observed problem, the observation may leave things out or distort them. Imagine a robot navigating with a broken sensor: it receives a noisy reading of its position, not the true position. That reading is an observation, and the true position is the state.

The practical consequence: when an agent only sees observations, it may have to act on incomplete information. Two different states might produce the same observation, and the agent can't tell them apart. That's not a failure of the agent—it's a structural property of the problem.

Common mistake: Treating an observation as "a state you can't fully see." The cleaner mental model is that the state is the full description and the observation is the input the agent actually works with. In fully observed problems, they coincide. In partially observed problems, the observation is a window onto a larger state.

Time and Structure: Timestep, Episode, Trajectory

The decision loop repeats, but how we chunk that repetition matters. Reinforcement learning problems come in two flavors, and the vocabulary changes depending on which one you're looking at.

A timestep is one single decision-and-response cycle. State observed, action taken, reward received, new state reached. That's one timestep.

An episode is a complete run from a starting state to a terminal state. One game of chess. One maze run from start to finish. One round of a video game. When the episode ends—checkmate, goal reached, game over—the agent resets and starts a new one.

A trajectory is the ordered sequence produced during that run: the states, actions, and rewards the agent experiences as it moves through time. If the episode is the container, the trajectory is the experience sequence inside it.

Here's a distinction that trips up beginners: episode and trajectory are often used interchangeably, and in many tutorials you'll see them treated as the same thing. That overlap is fine for episodic tasks. When the terms matter, remember the emphasis: an episode is a bounded interaction run, while a trajectory is the ordered sequence of states, actions, and rewards generated along the way.

Some problems don't have episodes at all. These are continuing tasks—they keep going indefinitely. A stock-trading agent doesn't reset at the end of a "game." It just keeps making decisions day after day. This distinction matters because many algorithms assume episodes. When you read a paper that talks about "episodic training," recognizing the structure tells you how to interpret what the algorithm is doing.

What the Agent Wants: Return, Discount, and Value

Here's the core tension in reinforcement learning: the agent receives rewards one at a time, but it's not trying to maximize any single reward. It's trying to maximize the return—the total accumulated reward over time.

Why does this distinction matter? Because rewards are delayed. A chess move that sacrifices a piece looks bad in the moment—negative reward—but it might set up a checkmate ten moves later. If the agent only chased immediate rewards, it would never learn to sacrifice. The return captures the whole picture.

But there's a problem with simply adding up all future rewards: the future is uncertain, and an infinite future produces an infinite sum. That's where the discount factor comes in. Usually written as gamma (γ), it's a number between 0 and 1 that shrinks the value of future rewards. A reward received now is worth its full value. A reward received one step in the future is worth gamma times its value. Two steps out, gamma squared. The further away the reward, the less it counts.

The discount factor isn't just mathematical convenience. It reflects a real preference: a reward now is more certain than the same reward later. Something might go wrong in between. The agent should care more about what it can actually influence soon than about distant possibilities that might never arrive.

Now we get to the term that confuses everyone: value. A value function is an estimate of the expected future return from a given state. In plain English: "If I start here and follow my current strategy, how much total reward should I expect?"

The key word is estimate. The agent doesn't know the future. It has beliefs about the future, built from experience, and those beliefs are its value estimates.

There's also an action-value function, often called the Q-value. Same idea, one twist: instead of asking "how good is this state?", it asks "how good is it to take this specific action in this state?" That distinction matters for decision-making. The agent doesn't just need to know that a position is strong; it needs to know which move creates that strength.

Common mistake: Treating reward, return, and value as the same thing. They're three different jobs. Reward is the immediate feedback after one action. Return is the accumulated total over time. Value is an estimate of what future return to expect. One is a signal, one is a sum, one is a prediction.

How the Agent Decides: Policy and Exploration

Knowing what's valuable is only half the problem. The agent also needs a rule for turning that knowledge into actions. That rule is the policy.

A policy is the agent's decision rule: given a state, what action should I take? It can be deterministic—always move right in this situation—or stochastic—move right 70% of the time and up 30% of the time. Stochastic policies sound odd at first, but randomness is sometimes the smart choice. If you're not sure which action is best, spreading your bets gathers information.

This brings us to one of the most important tensions in reinforcement learning: exploration versus exploitation.

Exploitation means choosing the action you currently believe is best. You've learned that moving right tends to work, so you move right. The problem? Your belief might be wrong. Maybe moving up would work even better, but you've never tried it enough to know.

Exploration means trying other actions to test your beliefs. Move up instead. See what happens. You might discover something better—or you might confirm that right was the right choice.

An agent that only exploits never discovers better options. An agent that only explores never uses what it has learned. Every real RL algorithm has to balance the two, and how it does that balance is one of the defining features of different approaches.

Common mistake: Confusing policy with value function. The value function is the estimate—"this state is probably worth 5.2 units of future reward." The policy is the decision rule—"given what I believe, I'll take action A." The value function informs the policy, but they're different objects. Think of the value function as the map and the policy as the route you choose based on it.

Terms You Will Meet in Algorithms: Model and Policy Learning

Once you start reading about actual algorithms, you'll encounter a few more terms that sound intimidating but have simple cores. Two distinctions do most of the work, and they answer different questions.

The first question: does the agent learn how the environment works? A model in RL is the agent's knowledge of environment dynamics—specifically, how states transition and how rewards are given. An agent with a model can plan: "If I take this action, I predict the environment will respond that way." Model-based methods use this knowledge to simulate and plan. Model-free methods skip the model entirely and learn directly from experience, updating their estimates based on what actually happened rather than what they predicted would happen.

The second question: who is the agent learning from? An on-policy method learns about the policy it's currently following. It learns from actions it took under its current strategy, so when the policy changes, the agent needs fresh experience. An off-policy method can learn about one policy while behaving according to another. It can study its own past actions from when it was behaving differently—or even another agent's actions—and still learn something useful.

These are separate axes. A method can be model-based or model-free, and independently on-policy or off-policy. When you see these terms in an algorithm description, ask which question each one answers: Does the agent use a model of the environment? and Is the agent learning from its current behavior or from other behavior?

Note: This article deliberately stops short of algorithm-specific vocabulary like Q-learning, policy gradients, or actor-critic methods. Those terms deserve their own treatment. For now, the goal is recognition: when you see "model-free" or "off-policy" in an algorithm description, you'll know which part of the learning process it's describing.

A Quick Lookup Table for the Terms That Get Confused

When you're in the middle of a tutorial and need a fast answer, use this table. It covers the pairs that trip up most beginners.

Term	What it really means	Its job in the loop
State	The information that describes the situation for the task	The full description
Observation	What the agent actually receives; may equal the state or be a partial/noisy version	The agent's input
Reward	Immediate feedback after one action	The signal
Return	Total accumulated reward over time	The objective
Value	Estimated future return from a state	The prediction
Q-value	Estimated future return from taking a specific action	The action-level prediction
Policy	The rule for choosing actions	The decision-maker
Episode	One complete run from start to finish	The unit of experience
Trajectory	The ordered sequence of states, actions, and rewards in a run	The experience sequence
Model-based	Uses knowledge of environment dynamics to plan	The planner
Model-free	Learns directly from experience without an environment model	The direct learner
On-policy	Learns about the policy it is currently following	The self-learner
Off-policy	Can learn from behavior generated by a different policy	The flexible learner

Putting the Map to Work

The next time you're reading an RL tutorial and hit an unfamiliar term, run it through a quick three-step check.

First, name its role. Is it about the loop itself—agent, environment, state, action, reward? Is it about time structure—timestep, episode, trajectory? Is it about the objective—return, discount, value? Is it about the decision rule—policy, exploration, exploitation?

Second, identify what goes in and what comes out. A value function takes a state and produces a number. A policy takes a state and produces an action. If you can name the input and output, you understand the term's mechanism.

Third, compare it with its nearest confusing neighbor. Reward versus return versus value. State versus observation. Episode versus trajectory. Policy versus value function. Each pair differs on one clear axis, and naming that axis is what makes the distinction stick.

Try it on a phrase like "discounted return." Role: objective. Input: a sequence of future rewards. Output: a single number. Nearest neighbor: plain return, which doesn't shrink future rewards. The discount factor is the only difference—and now the phrase isn't mysterious anymore.

Each group in this map has a deeper story. The reward-to-return pipeline has its own logic about why discounting matters and how reward design can go wrong. Policies have their own nuances about determinism and uncertainty. States and observations have their own complications when the agent can't see everything.

But you don't need all of that to decode your next tutorial. You need the map. Keep it handy, run each new term through the three-step check, and the vocabulary stops being a wall—it becomes a set of signposts pointing to the same learning process from different angles.