Partial Observability in Reinforcement Learning: When the Agent Cannot See the Full State

Imagine you are playing a card game against an opponent. You can see your own hand perfectly, but your opponent's cards are hidden. You watch them draw, discard, and wager. At some point, you face a decision: bet big or fold. The problem is that two very different worlds could be sitting across the table. Your opponent could be holding a winning hand and betting with confidence, or they could be bluffing with nothing. From your seat, both situations can look identical.

This is the gap at the heart of partial observability in reinforcement learning. The agent must act without knowing which hidden world it is really in.

The Same View, Two Different Worlds

In reinforcement learning, the state is the true situation of the environment. It is everything that matters for deciding what to do next. The observation is what the agent actually receives through its sensors.

In many beginner environments, these are the same thing. A grid-world robot knows its exact coordinates. A cart-pole agent receives the pole's precise angle and velocity. When observation equals state, the agent has everything it needs in a single snapshot.

But that equality breaks all the time in realistic settings. Consider a robot navigating a hallway with a camera. It approaches a corner. Behind that corner, there might be an open corridor or a solid wall. The camera sees the same blank corner either way. The robot's observation is identical, but the true state of the world is different, and the correct action depends on which world is real. If the corridor is open, the robot should keep moving forward. If the wall is there, it should turn around.

The observation did not lie. It just did not tell the whole story.

This is the plain-language distinction to hold onto: the state is the truth; the observation is what the agent sees. When they diverge, the agent faces hidden state reinforcement learning.

What Partial Observability Actually Means

When an agent's observations carry only part of the information about the true underlying state, the problem is called partial observability. The environment still follows a real, underlying process. The agent just receives a filtered or noisy view of it.

The formal name for this setting is the partially observable Markov decision process, usually shortened to POMDP. You do not need the equations yet. The concept matters more: a POMDP is a Markov decision process where the agent cannot directly observe the state. Instead, it receives observations that are related to the state but do not fully reveal it.

Think of the flow this way:

1. The environment has a true hidden state.
2. That state produces an observation the agent can see.
3. The agent must choose an action based on that observation.

In a standard MDP, the agent knows the state directly, so step 2 is trivial. In a POMDP, step 2 is lossy. Information is dropped between the state and the observation, and the agent has to make decisions despite that missing information.

This is why partial observability reinforcement learning feels different in practice. The agent is not just learning which action leads to reward. It is also trying to figure out where it is, with incomplete evidence.

Why a Memoryless Policy Breaks Down

A policy is a rule that turns what the agent knows into an action. In a fully observable environment, a policy can map the current state directly to an action. The agent looks at where it is and decides what to do next.

Now consider what happens when the agent only sees the current observation, and that observation is ambiguous. Two different true states can demand opposite actions while producing identical observations.

Here is a small worked example. Imagine a robot that must decide whether to charge its battery or keep working. The robot cannot see its own battery level directly. It only sees a warning light. The light is off in two situations: when the battery is fully charged, and when the battery is at 40 percent. If the battery is full, the robot should keep working. If the battery is at 40 percent, it should charge soon. The observation is the same, but the right action is different.

A policy that only looks at the current observation cannot resolve this. It sees the same input and must produce the same output. No amount of cleverness in a single-snapshot policy fixes the missing information. The agent is guessing, and guessing is not a strategy.

This is the core failure mode of partial observability. The policy is not wrong because it is badly designed. It is wrong because it is blind to the context it needs.

How Memory Restores the Missing Context

The fix is not to see better. The fix is to remember more.

Instead of acting on a single snapshot, the agent can keep a history: the sequence of past observations and actions it has accumulated. That history carries information the current observation lacks.

Think of a detective arriving at a crime scene. A single glance tells them something, but not enough. They need the accumulated clues: what witnesses said, what was moved, what time things happened. Each clue alone is partial. Together, they narrow the possibilities.

An RL agent can do the same thing. If the robot with the battery warning light remembers that it has been working for six hours, the off light means something different than if it just started. The history gives the observation meaning it did not have on its own.

But notice what memory actually does here. It does not let the agent see the battery level directly. It gives the agent better evidence about which hidden state is likely. The robot still does not know the battery is at 40 percent with certainty. It just has a more informed basis for acting.

That distinction matters. Memory preserves evidence from earlier moments; it does not guarantee certainty. A history can improve a decision without revealing the hidden state completely. Sometimes the evidence points clearly in one direction. Sometimes it stays genuinely ambiguous, and the best the agent can do is act on the balance of probabilities.

In practice, modern agents often learn this memory automatically rather than having it hand-designed. The agent's internal memory condenses the history into a summary it can actually use at decision time. The details of how that works involve recurrent networks and other machinery you will meet later. For now, the concept is what matters: memory turns a sequence of incomplete snapshots into useful context.

Beliefs: When the Agent Tracks Probabilities Instead of Certainty

There is another way to handle hidden information, and it is more elegant than storing every past detail.

Instead of remembering the full history, an agent can maintain a belief: a probability distribution over the possible hidden states it might be in. Each possible state gets a probability, and those probabilities add up to one.

Each new observation updates that belief. When the agent sees something consistent with a particular hidden state, that state becomes more probable. When it sees something inconsistent, that state becomes less probable.

Return to the card game. At the start, you have no idea what your opponent holds. Every hand is roughly equally likely. Then they bet aggressively. That action is more consistent with a strong hand than a weak one, so your belief shifts. Then they hesitate before calling. That hesitation is evidence too. Your belief shifts again.

Here is the key point: your belief is not certainty. It can stay broad when evidence is weak. It can shift when new evidence arrives. It can even widen if later actions make several hands plausible again. You never know for certain what they hold. What you have is a sharper estimate, and that estimate is what guides your decision.

A belief is more compact than a full history. You do not need to remember every action your opponent took. You just need the current probability distribution over their possible hands. And crucially, a policy can act on that distribution. Just as a fully observable policy maps a known state to an action, a belief-based policy maps the current probability distribution to an action. The agent chooses the action that makes sense given what it currently considers possible.

The tradeoff is that maintaining a belief requires you to model how observations relate to hidden states. You need to know how likely each observation is given each state.

This belief-state idea is central to POMDP reinforcement learning. It gives the agent a principled way to act under uncertainty rather than pretending the uncertainty does not exist.

When Partial Observability Matters (and When It Does Not)

Partial observability is everywhere in real systems. Noisy sensors, occlusions, hidden opponent information, delayed feedback, missing measurements. If you build an agent that interacts with the physical world, other agents, or any system with imperfect sensing, you will encounter it.

Here is a practical test for deciding whether partial observability matters in your problem: Can two different true states produce the same observation while requiring different actions? If yes, a memoryless policy will struggle. If no, treating the observation as the state might work fine.

Notice the second half of that question. It is doing real work. Two hidden states can share an observation and still demand the same action. In that case, the ambiguity does not matter for the decision. The agent does not need the full physical state. It only needs enough decision-relevant information to choose well.

Many beginner environments are fully observable by design. CartPole gives you the pole's angle. Grid worlds give you exact positions. In those settings, you do not need memory, and adding it will not help. The observation genuinely contains everything the agent needs.

But the moment you move to richer problems, the assumption breaks. A robot with a camera does not see its own position. A game-playing agent does not see the opponent's strategy. A trading agent does not see the market's hidden state. These are partially observable problems, and treating them as fully observable will cap your agent's performance.

My rule is simple: ask the two-states-one-observation question early. If the answer is yes, plan for memory from the start. If the answer is no, do not add complexity you do not need.

A Mental Model to Keep

Here is the durable contrast to carry forward: the state is the truth, the observation is what the agent sees, and when they diverge, the agent needs memory or belief to decide well.

The one-line decision rule: whenever two different hidden states could produce the same observation while demanding different actions, a single-snapshot policy will fail. Add memory or a belief estimate.

Try a small self-experiment. Take an environment you already know, or one you plan to use, and ask the diagnostic question. Can two different true states produce the same observation? If they can, do those states require different actions? You will find that some environments pass the test easily, and others fail it in ways you did not expect.

That habit, asking where the information gap is, will serve you better than memorizing any algorithm. Partial observability is not a rare edge case. It is the default condition of acting in a world you can never fully see.