Recurrent Policies and Memory in Reinforcement Learning

A memoryless policy keeps repeating the same mistake because each observation hides the information it needs. The reflex is to "just add an RNN." The real question is whether your agent needs to remember something it has seen, or whether it never saw the right thing at all.

When One Observation Is Not Enough

Imagine an agent navigating a corridor with two identical doors. Behind one door sits a reward; behind the other, a penalty. The agent opens a door, receives the outcome, and the episode resets. The catch: the doors look identical, and the reward location is fixed for the entire run.

A policy that maps observation to action—the standard feedforward setup—cannot solve this. Every observation looks the same, so every observation demands the same action. The agent opens the same door forever, never learning which one pays.

This is the partial-observability problem: the observation fails to identify the true state of the world. But partial observability explains why observations fail. This article is about the policy-side fix: giving the policy a mechanism to condition its decisions on what happened earlier.

Here is the reframe that matters. Memory is not an add-on feature you bolt onto a policy. Adding memory changes what the policy conditions on. A feedforward policy conditions on the current observation alone. A history-dependent policy conditions on the current observation and a summary of everything that came before it.

What a Recurrent Policy Actually Computes

A recurrent policy replaces the mapping "observation → action" with a loop. At each timestep, the policy takes two inputs: the current observation and an internal hidden state. It produces two outputs: an action and an updated hidden state.

hidden_t = f(hidden_{t-1}, observation_t)
action_t = g(hidden_t)

The hidden state is not a transcript. It is a learned summary—a compressed vector that encodes whatever the network has learned to preserve from the interaction history. The recurrence is what lets information from timestep 3 influence a decision at timestep 12. Without the loop, that information is gone the moment the observation leaves the input.

If you unroll this loop in time, you get a chain: each step feeds the previous hidden state forward into the next computation. This unrolled structure is why recurrent policies are trained with backpropagation through time—gradients must flow backward through the same chain to reach the parameters that produced early hidden states.

The practical implication: the policy's decision at any moment depends on the entire path of observations and actions that led there, not just the current frame. That is the entire point.

What the Hidden State Can and Cannot Remember

The hidden state is lossy compression. It must be: a fixed-size vector cannot preserve every detail of an arbitrarily long interaction history. The network learns what to keep, and that learning is not guaranteed to succeed.

Three failure modes follow directly.

Forgetting. The hidden state can drop information from the distant past, especially when long stretches of irrelevant observations intervene. An LSTM or GRU mitigates this with gating mechanisms, but mitigation is not elimination.

Conflation. Two different histories can produce similar hidden states. If the policy needs to distinguish them, it may fail even though the information was technically "in memory" at some point.

Unlearned relevance. The network must discover which historical details matter. Nothing tells it explicitly. If the credit-assignment problem is hard—if the relevant event happened many steps before the reward—the network may never learn to preserve the right information.

Contrast this with an explicit belief state: a probability distribution over hidden states, maintained by Bayesian updates. A belief state preserves information in a principled way, but it requires knowing the transition and observation models. A recurrent policy learns its summary from data, which is why it works when the environment model is unknown—and why it can fail when the learning problem is too hard.

Memory Failure vs. a Lossy Observation Design

Here is the diagnostic that separates productive debugging from wasted effort: when a recurrent policy underperforms, decide whether the problem is the memory mechanism or the observation feeding it.

The rule is blunt. If the needed information never appears in any observation, no amount of recurrence can recover it. Recurrence can only summarize what the agent has seen. It cannot manufacture a signal that was never observed.

If the information is present but spread across time—the door color appears only when you approach it, the reward location is revealed only after you commit—then recurrence is the right tool. The policy needs to carry that information forward.

The practical test: ask whether a human or an oracle with access to the full history could solve the task. If the answer is no, fix the observation first. If the answer is yes, and your memoryless policy fails, then memory is the missing ingredient.

Common mistake: Adding recurrence to a fully observable task. If every observation already contains the full state, a recurrent policy adds cost, complexity, and training instability without adding capability. Memory is for tasks where history carries information the present hides.

The Practical Cost of Recurrent Policies

Recurrent policies are harder to train than their feedforward counterparts. The costs are not exotic—they are operational, and they compound.

Sequence boundaries. The hidden state must reset when an episode ends. If trajectories leak across each other within a training batch, the policy learns from blended histories that never occurred. This sounds like an implementation detail; it is a correctness requirement.

Backpropagation through time. Gradients must flow backward through the unrolled sequence, which raises the computational cost of each update and introduces temporal credit-assignment and gradient-propagation issues that feedforward networks do not face.

Experience replay becomes delicate. Off-policy methods assume individual transitions are independent samples. A recurrent policy breaks that assumption: a single transition is meaningless without the sequence that preceded it. Replay must store and sample sequences, not transitions, and the hidden states used during collection may be stale by the time they are replayed.

These costs are justified only when the task genuinely needs history. When it does not, you are paying for memory that the policy cannot use.

A Decision Rule for Adding Memory

Before you reach for an RNN, run this checklist:

1. Is the observation ambiguous? Does the same observation correspond to different states that require different actions?
2. Does the needed signal appear across time? Is the information present in the observation stream, just not all at once?
3. Is a memoryless baseline failing for a reason recurrence would fix? If the baseline fails because the observation lacks the signal entirely, recurrence will fail too.

My rule is simple: fix the observation or representation first, add memory only when history is genuinely the missing ingredient.

Before committing to a recurrent policy, consider cheaper alternatives. Frame stacking—feeding the last k observations as input—gives the policy a short window of history at a fraction of the cost. Richer features can sometimes make the relevant information visible in a single observation. An explicit belief state is the right choice when you have a model of the environment dynamics.

Memory is a tool for summarizing history. It is not a patch for a bad observation design. When your recurrent policy fails, the first question is not "how do I make the memory work better?" It is "was the information ever there to remember?"