Belief States in POMDPs: Acting on Probabilities About Hidden State

A policy that acts on the latest observation alone is guessing. A belief state is how you stop guessing.

Why a Single Observation Is Not Enough

In a partially observable environment, the observation is a projection of the hidden state, not the state itself. Two different hidden states can produce the same observation while demanding different actions. That single fact breaks any policy that keys its decisions to the raw observation.

Consider a simple grid world where an agent sees only what is directly in front of it. The agent stands in a corridor. The observation is identical whether the corridor leads to a reward or to a dead end. A memoryless policy—one that maps observation to action—cannot distinguish the two cases. It will make the same choice in both, and it will be wrong half the time.

The problem is not that the agent receives too little information. The problem is that the agent throws away information it already has. The history of past observations and actions contains clues that the latest observation alone does not. If the agent saw a reward earlier in the episode, or heard a door close behind it, that context changes what the current observation means.

The fix is to carry more than the latest observation. The agent must carry a summary of everything it has seen and done—a compressed record of its entire interaction with the environment. That summary is the belief state.

The Belief State: A Distribution, Not a Guess

A belief state is a probability distribution over the possible hidden states, conditioned on the full history of observations and actions. If the hidden state space has three states, the belief is a vector of three probabilities that sum to one:

b(s) = P(s | history of observations and actions)

Each component answers the question: given everything the agent has seen and done, how likely is it that the true state is s?

The crucial move is refusing to collapse this distribution into a single guess. A point estimate—"I am in state 2"—throws away confidence information. The belief keeps every possibility alive, weighted by how plausible it is. This matters because the right action often depends on the shape of the distribution, not just its peak. If two states are nearly equally likely but demand opposite actions, the agent should act in a way that is robust to both possibilities, not commit to the slightly more probable one.

The belief is also a sufficient statistic. It captures everything from the history that matters for predicting future reward. Two different histories that produce the same belief are indistinguishable from the perspective of future decision-making. This is what makes the belief a legitimate replacement for the raw history: nothing is lost in the compression.

How Actions and Observations Update the Belief

The belief does not sit still. Every action and every new observation reshapes it. The update happens in two steps: predict, then correct.

Predict. After the agent takes an action, the world may move. The belief must be propagated forward through the transition dynamics. Probability mass spreads from each state to the states the world could have moved to. If the agent is 70% sure it is in state A and action move-left leads from A to B with probability 0.9, then some of that 70% mass now sits on B.

Correct. When the new observation arrives, the belief is reweighted by the likelihood of that observation under each possible state. States that make the observation likely gain probability. States that make it unlikely lose probability. The result is the posterior belief.

The recursion has a compact form:

b′(s′) ∝ P(o | s′) × Σₛ P(s′ | s, a) b(s)

The action spreads probability according to where the world could move. The observation then concentrates probability on the states consistent with what the agent actually saw. This is exactly Bayesian filtering, and it is the same mechanism used in Kalman filters and particle filters. The belief update requires knowing the transition model and the observation model—the agent must know how the world moves and how observations are generated from states.

Why the Belief Makes the Problem Markov

Here is the central insight that makes POMDPs tractable as a conceptual framework: the belief state satisfies the Markov property even when the underlying hidden state does not.

The future depends on the past only through the current belief. Given the belief, the history that produced it carries no additional information about what happens next. This means value functions and policies can be written as functions of the belief rather than of the raw history. The belief is the state of a new, fully observable decision process.

This is why belief state reinforcement learning works. The partially observable problem becomes a fully observable MDP over a larger state space—the space of all probability distributions over the hidden states. Value functions over beliefs inherit the structure of value functions over the underlying states. The value of a belief is a weighted combination of the values of the underlying states, where the weights come from the belief itself.

The tradeoff is immediate and unforgiving. The belief space is continuous, and its dimensionality grows with the number of hidden states. A hidden state space of ten discrete states produces a nine-dimensional simplex of beliefs. Exact belief MDPs are rarely tractable beyond small problems.

When the Belief Model Helps and When It Breaks Down

The exact belief state is the right mental model when three conditions hold: the state space is small and discrete, the transition and observation models are known, and the distribution can be represented and updated exactly. This is the regime of classical POMDP solvers and small diagnostic problems.

Outside that regime, the exact belief becomes expensive or impossible. The costs stack quickly:

• Model requirement. The belief update needs accurate transition and observation probabilities. In most real environments, these are unknown and must be learned.
• Dimensionality. The belief space grows with the number of hidden states. Continuous state spaces make exact updates intractable.
• Relevance. The belief tracks every state in the environment, including states irrelevant to the task.

Practice offers two escape hatches. The first is particle filtering: represent the belief as a set of weighted samples rather than a full distribution. Particles scale to larger and continuous state spaces, at the cost of approximation error. The second is learned belief-like representations: recurrent networks that compress the history into an internal state without ever computing an explicit posterior. These scale to high-dimensional observation spaces and need no environment model, but they trade away the guarantees of exact Bayesian filtering.

The distinction between what is known and what is inferred matters here. Exact Bayesian filtering is a well-understood mechanism with clean mathematics. Learned belief representations are an active research area with less settled guarantees about what their internal states actually track.

A Mental Model for Reading RL Papers on Beliefs

When you encounter a paper or system that claims to use belief states, run it through a short checklist:

1. Does the method assume a known model, or does it learn the belief from data? Exact Bayesian filtering requires the transition and observation models. Learned methods infer a belief-like representation from experience.
2. Is the belief exact or approximate? A full posterior over the state space is exact. A particle set or a factored distribution is approximate.
3. Is it a full distribution or a compressed summary? A recurrent network's hidden state may encode task-relevant information without representing a calibrated posterior over all states.

The trap is conflating "belief-like" with "exact belief." Many modern methods learn internal representations that behave like beliefs—they track uncertainty, integrate evidence over time, and improve decisions under partial observability—without being calibrated posteriors. A recurrent network trained end-to-end may learn a representation that is useful for the task but wrong as a probability estimate. That distinction matters when you need to reason about confidence, robustness, or failure modes.

The belief-state mental model is the reference point against which all approximate methods should be judged. It tells you what an ideal information state would look like, even when you cannot compute it.

The Practical Takeaway

The belief state converts a question about hidden state into a question about information. Instead of asking "where am I?", the agent asks "what distribution over locations is consistent with everything I have seen and done?"

For any partially observable problem, ask three questions:

1. What is the hidden state? Define the quantities the agent cannot observe directly but needs to act well.
2. What distribution over that state would I need? Identify the belief that would make the problem Markov.
3. Can I maintain that distribution exactly or only approximately? Choose between exact Bayesian filtering, particle methods, and learned representations based on the state space, the availability of a model, and the computational budget.

The belief state is the ideal that approximate methods approximate. Keep it as your reference model, and you will be able to see clearly what any learned or sampled representation is actually buying you—and what it is giving up.