advanced
8 min read
Belief States in POMDPs: Acting on Probabilities About Hidden State
A policy that acts on the latest observation alone is guessing. A belief state is how you stop guessing.
Read tutorialconcept
States and Observations
State representations, observations, Markov sufficiency, hidden information, and the information available to an agent when choosing actions.
Tagged articles
States and Observations articles
21 articles in this tag.
advanced
11 min read
Centralized Training and Decentralized Execution in Multi-Agent RL
The agent that acts at deployment is not the agent that learns. Training sees a richer world than execution ever will.
Read tutorial
intermediate
7 min read
Credit Assignment in Reinforcement Learning: Connecting Late Rewards to Earlier Actions
The agent makes a move. Nothing happens. It makes another move, then another. Twenty steps later, a reward arrives—and the agent has no idea which action…
Read tutorial
advanced
9 min read
Formalize Information Constraints in Centralized Training and Decentralized Execution
Your team scores well in the training harness. You strip the centralized critic, deploy the actors, and the score collapses — even though the environment…
Read tutorial
advanced
10 min read
Audit Centralized Training and Decentralized Execution in a Toy Task
A CTDE run can post a beautiful training curve while the deployed policy quietly cheats. The critic saw the joint state. So did the actor. Nobody noticed,…
Read tutorial
intermediate
8 min read
Should This Project Use Deep RL? Practice Choosing With Project Scenarios
You have the checklist memorized. State size. Data cost. Observability. Action structure. Baselines. Debugging burden. Then a real project lands on your…
Read tutorial
beginner
13 min read
How to Design a Reinforcement Learning Environment
Most beginners assume the algorithm is the hard part of reinforcement learning. Pick the right one, tune it well, and the agent will learn. Then they spend…
Read tutorial
intermediate
12 min read
Markov Decision Processes: The Formal Map Behind Reinforcement Learning
The agent-environment loop feels simple until you try to make an algorithm learn. Then the loop stops being enough. The agent needs to know what it can…
Read tutorial
advanced
13 min read
Markov Games and Moving Learning Targets in Multi-Agent RL
The game never changes. Your experience of it changes every time another agent updates.
Read tutorial
intermediate
9 min read
Markov State Sufficiency in Reinforcement Learning: When Is State Enough?
Two situations look identical to your agent. It takes the same action. One time the world rewards it; the next time the world punishes it. You blame the…
Read tutorial
beginner
10 min read
Practice Formulating a Small Markov Decision Process in Python
You understand what an MDP is. Then someone hands you a task description and a blank file, and the understanding evaporates.
Read tutorial
advanced
11 min read
State Abstraction in MDPs: Bisimulation and Safe State Merging
Merging states is not compression. It is a claim about the future, and most merges quietly change the problem.
Read tutorial
advanced
8 min read
What One Bit of Communication Enables in a Two-Agent Coordination Game
Two agents want the same thing. Both know the rules. Both are fully cooperative. And they still fail half the time — until someone is allowed to say a…
Read tutorial
beginner
10 min read
Partial Observability in Reinforcement Learning: When the Agent Cannot See the Full State
Imagine you are playing a card game against an opponent. You can see your own hand perfectly, but your opponent's cards are hidden. You watch them draw,…
Read tutorial
advanced
10 min read
Derive the Bellman Control Equation Over POMDP Beliefs
You write down the MDP Bellman equation, swap $s$ for $b$, and the result looks plausible. It also predicts nothing you can act on. The symbol changed; the…
Read tutorial
intermediate
12 min read
Deriving the Bayesian Belief Update for a POMDP
The agent never sees the state. It sees a flicker of evidence, acts, and sees another flicker. Yet the entire decision problem collapses into a fixed-size…
Read tutorial
beginner
9 min read
Practice POMDP Belief Updates With a Small Python Example
You can recite the belief update formula and still have no idea whether you are applying it correctly. The way out is not more reading. It is a two-state…
Read tutorial
advanced
7 min read
Recurrent Policies and Memory in Reinforcement Learning
A memoryless policy keeps repeating the same mistake because each observation hides the information it needs. The reflex is to "just add an RNN." The real…
Read tutorial
intermediate
10 min read
Compare Memoryless and Recurrent Policies in a Partially Observed Task
A memoryless agent can learn perfectly and still lose. It extracts every drop of signal from the observation it receives, then plateaus at a score a coin…
Read tutorial
absolute beginner
13 min read
Reinforcement Learning Terms: A Practical Reference for States, Rewards, Values, and Episodes
You're halfway through a tutorial when a sentence stops you cold: "The agent updates its Q-value using the discounted return from the trajectory." You know…
Read tutorial
beginner
14 min read
States, Observations, and Actions in Reinforcement Learning
The agent never acts on the world. It acts on a report about the world—and that report is often incomplete, noisy, or misleading. Understanding the gap…
Read tutorial