States, Observations, and Actions in Reinforcement Learning

The agent never acts on the world. It acts on a report about the world—and that report is often incomplete, noisy, or misleading. Understanding the gap between the two is the difference between building an agent that makes good decisions and debugging one that fails in ways you cannot explain.

Picture a delivery robot rolling down a hallway. Its camera points forward. The robot sees the corridor ahead: the doors, the turns, the obstacles in its path. But it cannot see what is behind it. It cannot see around the corner. It cannot see the person about to step out from a doorway to its left.

When that robot makes a decision, what is it actually deciding from? The complete situation of the hallway, or the partial picture its camera captured?

If you said "the partial picture," you are already ahead of most beginners. The most common misconception in reinforcement learning is that the agent sees the world the way you see a video game screen—complete, clear, and fully available. It does not. The agent receives a report, and that report is often incomplete, noisy, or misleading.

The Agent Never Sees the World—Only a Report About It

Let's make the vocabulary precise, because these two words get conflated constantly.

The state is the complete description of the environment at a moment in time. It is everything that is true about the world right now: every object's position, every relevant physical property, every hidden condition that could influence what happens next. In the hallway example, the state includes the robot's exact location, the layout of every room, the people moving through the building, and the obstacles that exist even when no camera can see them.

The observation is what the agent actually receives. It is a partial, possibly noisy report about the state. The robot's camera frame is an observation. It contains some information about the state—what is directly in front of the robot—but it omits everything else.

Here is the practical consequence that matters: the agent must decide from the observation, not from the state. The robot cannot act on information its camera never captured. It cannot swerve around a person it cannot see. It cannot plan around a closed door that is behind it.

This sounds obvious when stated plainly, but beginners violate it constantly. They design an agent's inputs as if the agent had god's-eye access to the full situation. Then they wonder why the agent makes choices that look irrational from the outside. The agent was not irrational. It was blind to part of the problem.

Think of it as a builder's constraint: you can only act on the information that actually reaches you. If you are building the agent, you are responsible for deciding what information reaches it—and for knowing what you left out.

State Space and Observation Space: What the Words Actually Mean

Once you accept the state-observation distinction, the next step is naming the full range of possibilities.

The state space is the set of all possible states the environment can be in. The observation space is the set of all possible observations the agent can receive.

These two sets can be very different in size and character. Consider a few examples:

Problem	What the state includes	What the agent observes
Grid navigation	The agent's exact (x, y) coordinates	The same coordinates, if fully observed
Chess	The position of every piece on the board	The same board position
Robot arm	Joint angles, velocities, object locations	Sensor readings that may be noisy or incomplete
Atari game	Full game state, including off-screen objects	The current pixel frame on the screen

Notice the pattern. In some problems—like chess or a simple grid—the observation carries everything the agent needs. In others—like the robot arm or the Atari game—the observation is a projection of the state. It captures some information and discards the rest.

Many beginner RL setups treat state and observation as identical. The environment hands the agent a complete description of its situation, and the agent acts on that description. This is convenient for learning, but it hides the distinction. When you move to real problems—robotics, autonomous driving, any system with sensors—the distinction stops being academic. It becomes the difference between an agent that works and one that fails mysteriously.

Here is a plain-language test you can apply to any problem: if you can name information the agent needs but does not receive, the observation space is not the state space. Run that test on your own problem. If you can list even one missing piece of information that would change the agent's best decision, you are dealing with partial observation.

Fully Observed vs Partially Observed Environments

When the observation carries everything the agent needs to know about the state, we call the environment fully observed. When it omits or distorts information the agent needs, we call it partially observed.

Chess is the cleanest example of a fully observed environment. The board position you see is the complete state. Nothing is hidden. Every piece is visible, every rule is known, and the best move depends only on what is currently on the board.

Now contrast that with a robot that has only a forward camera. The robot might need to know whether a person is approaching from behind, whether a door has opened to its side, or whether an obstacle lurks just around the corner. None of that appears in the camera frame. The environment is partially observed because the observation omits information that matters for the robot's decisions.

Why does this distinction matter so much? Because partial observability forces the agent to do something fundamentally harder than react. When the observation is incomplete, the agent must remember or infer what it cannot see. It must track history—what it has observed over time—to build a better picture of the true state.

Consider the most dangerous case: the same observation can arise from two different true states that demand different actions. A robot at an intersection sees the same camera frame whether a pedestrian is about to step into its path or has already crossed and is standing safely on the far side. The observation is identical. The correct action is not. The robot cannot decide correctly from the current observation alone. It needs more information—more history, another sensor, or a model of how pedestrians behave.

This is the seed of a topic that will return later in your RL journey: some problems need memory or belief tracking, not just reaction to the current input. For now, the conceptual takeaway is enough: when observations are incomplete, the agent's job changes from "respond to what you see" to "figure out what is really going on."

Actions and Action Spaces: The Choices the Agent Can Make

The agent does not just observe. It acts. And the set of actions available to it shapes everything about the learning problem.

An action is one decision the agent makes at a time step. Move left. Play a chess piece. Apply a torque to a joint. The action space is the set of all actions available to the agent.

Action spaces come in two broad flavors, and the difference matters enormously for how you approach the problem.

Discrete action spaces have a finite number of distinct choices. Move left or right. Jump or duck. Choose one chess move from the legal options. These are the easiest action spaces to work with because the agent's decision is a selection from a countable list.

Continuous action spaces have an infinite range of possible values. Steering angle for a car. Torque for a robot joint. Throttle position for a drone. The agent is not choosing from a list; it is producing a value from a continuous range, and the range itself is part of the problem.

The size and type of the action space changes how hard the learning problem is. A discrete space with two actions is comparatively simple: the agent just needs to learn when each action is better. A continuous space with many degrees of freedom is much harder: the agent must learn not just which action to take but what value to produce, and the space of possibilities is infinite.

One more wrinkle: in some problems, the available actions depend on the current state. A chess player cannot move a piece that is not on the board. A robot cannot open a door that is locked. We sometimes write this as the set of valid actions in a given state. The action space is not always a fixed menu; it can change as the environment changes.

Here is the builder's rule of thumb I keep coming back to: the action space defines the range of what the agent can even try. If an action is not in the space, the agent cannot attempt it, cannot learn from it, and cannot discover whether it would have been useful. The action space sets the ceiling on what the agent can learn. Design it too narrowly, and you have removed the agent's best options before learning even begins.

Why the State-Observation Gap Breaks Decisions

Let's make the stakes concrete with a small scenario.

Imagine a robot navigating a warehouse. Its job is to reach a charging station before its battery dies. The robot observes its location through a GPS sensor. The true state includes the robot's exact position, the battery level, and the locations of obstacles. The observation includes the GPS reading and the battery level.

Now imagine the GPS sensor has noise. The robot is actually at position (10, 5), but the sensor reports (10.3, 5.1). The true state is unchanged—the robot has not moved—but the observation is slightly wrong. If the robot is close to a wall or a shelf, that small error could be the difference between a clear path and a collision. The agent acts on the observation, not the state. The noise in the report becomes noise in the decision.

This is not an exotic edge case. Real sensors are noisy. GPS readings drift. Cameras have glare. Lidar misses objects in fog. The gap between state and observation is not a theoretical curiosity; it is the everyday reality of building agents that interact with the physical world.

Here is the decision rule I want you to internalize: ask whether the observation contains everything needed to pick the best action. If the answer is yes, the problem is fully observed, and the agent can safely react to its current input. If the answer is no—if there is information the agent needs but cannot see—the problem is partially observed, and the agent must do something more sophisticated.

This distinction matters for every RL method you will learn later. Value methods and policy methods both assume the agent can act on what it observes. When the observation is incomplete, those methods need modification. They need memory. They need belief states. They need to track history.

There is a formal idea hiding behind this intuition, and it is worth naming now because you will meet it constantly: the Markov state. A state is Markov—or has the Markov property—when it captures everything relevant to the next decision. When the agent knows the Markov state, it does not need history to act well. The current state is sufficient. When the observation is not the full state, the agent loses this property. It cannot just look at the present; it must remember the past to understand the present.

For now, hold onto the simple version: a state is useful when it contains everything the agent needs to make the next decision well. If the agent needs history to decide correctly, the state representation is incomplete.

A Simple Way to Represent Any RL Problem

You now have the vocabulary. The next step is turning it into a habit you can apply to any problem you encounter.

Whenever you meet a new RL problem—in a paper, a library, a course exercise, or your own project—run this checklist:

1. Name the true state. What is the complete description of the environment at a moment in time? What information would you need to predict what happens next?
2. Name the observation. What does the agent actually receive? Is it the full state, or a partial, noisy report?
3. Name the action space. What choices can the agent make? Are they discrete or continuous? Do the available actions depend on the state?
4. Name the reward. What signal tells the agent whether it is doing well?

Let's run the checklist on a familiar problem: teaching an agent to play chess.

The true state is the position of every piece on the board, plus whose turn it is. The observation is the same board position—chess is fully observed. The action space is the set of legal moves available in the current position, which changes every turn. The reward is +1 for a win, −1 for a loss, and 0 for a draw.

Now run it on a harder problem: teaching a self-driving car to navigate a city street.

The true state includes the car's exact position and velocity, the positions and velocities of every other vehicle and pedestrian, the state of traffic lights, road conditions, and a thousand other factors. The observation is what the car's sensors capture: camera frames, lidar point clouds, GPS readings, radar returns. The observation is dramatically smaller than the state. The action space includes steering angle, throttle, and brake—all continuous. The reward might reward reaching a destination safely and quickly while penalizing collisions and traffic violations.

Notice what the checklist reveals. In chess, the observation is the state, so the agent can react to what it sees. In driving, the observation is a tiny slice of the state, so the agent must infer, remember, and predict. The checklist does not just name the components; it exposes the difficulty of the problem.

The most common beginner trap is conflating observation with state when writing down a problem. You see the environment hand the agent a vector of numbers, and you assume that vector is the complete situation. Run the checklist and ask the hard question: is there anything the agent needs to know that is not in that vector? If there is, you have found the partial observability that will shape your entire approach.

This explicit representation is the foundation for the Markov decision process, the formal framework that organizes all of RL. It is also the foundation for value methods and policy methods, which you will meet in later articles. Get comfortable naming the four components now, and every later topic will be easier to place.

Your Next Step

Take one problem you care about—a game you want to master, a robot you want to build, an automation task you want to solve—and run the checklist on it. Write down the true state. Write down what the agent actually observes. Write down the action space. Write down the reward.

Then ask the question that matters: where does the observation stop being enough?

If you can find a place where the agent needs information it does not receive, you have found the hard part of your problem. You have also found where you will need to get creative—adding memory, adding sensors, or redesigning what the agent observes.

The durable takeaway is this: the agent never acts on the world. It acts on its report of the world. The quality of that report decides whether the agent can learn well, no matter how clever your learning algorithm is. Build the report carefully, and you give the agent a chance. Build it carelessly, and no algorithm can save you.

The Markov decision process—the formal home of these ideas—is where we will go next.