Offline Reinforcement Learning Explained: Learning From a Fixed Dataset

An offline RL agent can report a sky-high value for an action it never actually tried. The critic is extrapolating into territory the dataset never covered—and without a live environment to correct the guess, the agent may chase that phantom value straight off a cliff.

If you've trained agents online, you carry a mental model where the environment acts as a teacher. The agent proposes an action, the environment answers with a reward and a next state, and that answer corrects the value estimate in real time. Bad guesses get punished. Hallucinated values get exposed. Remove live interaction, and you remove that correcting voice entirely. That single change transforms the learning problem—not because the math looks different, but because nothing is left to say "no."

What Changes When the Environment Goes Quiet

In online reinforcement learning, data collection and policy improvement are one continuous loop. The agent acts, observes, updates, and acts again. Every mistake generates fresh experience that pushes value estimates back toward reality.

Offline reinforcement learning—also called batch reinforcement learning—severs that loop. The agent learns from a fixed dataset of transitions, each stored as a (state, action, reward, next state) tuple. No new interaction happens during training. The dataset was collected by some behavior policy—the policy that generated the logs—which is usually unknown and may be a human, a hardcoded controller, another agent, or a mixture of policies. Its quality and coverage shape everything downstream.

This builds on the model-free versus model-based distinction you already know, but the axis has shifted. A model-based agent plans using a learned transition function. An offline agent doesn't plan at all—it must extract a policy from static records of someone else's decisions.

The real shift is this: in online RL, the environment is a teacher that punishes bad guesses. Offline RL removes that teacher, so the dataset must carry all the correction. If the data never witnessed a particular action, the agent has no evidence about what that action leads to—only whatever its function approximator chooses to invent.

Dataset Coverage: The Map Is Not the Territory

A fixed dataset records what the behavior policy actually did, not everything that was possible. For most states, most actions never appear. Coverage describes how much of the reachable state-action space those logged transitions represent—and sparse or narrow coverage leaves large silent regions.

Think of the dataset as a map drawn only where someone has walked. The agent must decide where to go next using a map with blank regions. In the covered areas, the contours are reliable. Outside them, the agent is guessing at terrain it has never seen.

Coverage quality depends on the behavior policy's diversity, not just its performance. A great but narrow policy can produce a dataset that is surprisingly hard to learn from. If a skilled human operator always takes the same route through a control task, the dataset will contain rich experience along that route and nothing anywhere else. The agent learns to imitate competence within a corridor—and has no grounding for what lies beyond it.

This is why offline RL datasets are judged by their coverage as much as their quality. A dataset full of excellent trajectories that all visit the same states may be less useful than a noisier dataset that explores broadly. The first teaches the agent one path. The second teaches it the shape of the space.

Support Is State-Conditional: Seeing a State Is Not Seeing an Action

Here is where the spatial metaphor can mislead you. "Coverage" sounds like a single region you can draw on a map. In practice, support is conditional: the dataset may contain a state many times, yet contain almost nothing about the specific action you want to evaluate at that state.

Consider a discrete control task with two actions at a critical state. The behavior policy chose action A there in hundreds of logged transitions, and action B only once—or never. The state is well covered. The action is not. If your critic assigns a high value to action B at that state, that estimate is not grounded in evidence; it is a generalization across a gap.

The problem sharpens in continuous action spaces, where the agent proposes actions from a continuous range. The dataset will essentially never contain the exact action the policy selects. Support becomes a question of proximity: is the proposed action close enough to logged actions at similar states that the critic's estimate is interpolation rather than invention? That boundary is rarely crisp. It is a density question, and the answer is usually an approximation.

This distinction matters because the policy's next-step distribution depends on it. A single unsupported action can move the agent into a state the dataset never covered, and from there every subsequent estimate is built on sand. The agent does not just leave the support of one action—it leaves the support of the state distribution entirely.

Extrapolation Error: Why the Critic Lies About Unseen Actions

Here is where the trouble starts. With function approximation, actions not present in the dataset can be assigned overly optimistic values—especially through bootstrapping. The critic generalizes from nearby data points and confidently fills in a value for an action it has never observed.

In online RL, this inflation gets corrected quickly. The agent actually tries the action, observes the true outcome, and the value estimate snaps back toward reality. The environment is a referee that overrules bad calls.

Offline RL has no referee. The inflated value propagates through bootstrapping: the critic's estimate feeds into the target for the next update, which inflates neighboring estimates, which inflates the policy's preference for those actions. The policy chases the phantom high value, and performance can collapse to near-random.

This is extrapolation error—the critic generalizing confidently into regions the data never supported. It's why naively treating a fixed dataset as a replay buffer for an off-policy algorithm usually fails. Off-policy methods assume that bad actions will eventually be tried and corrected. In the offline setting, that assumption is dead on arrival.

The failure is not that the critic is stupid. It's that the critic is doing exactly what function approximation does—generalizing—in a setting where generalization has no safety net.

Why a High Estimated Value Is Not a Supported Action

A high Q-value is a prediction about return, not evidence that the action was ever tried. The estimate can be high for two very different reasons: the action is genuinely good, or the critic is extrapolating into an unsupported region. The agent cannot reliably tell which is which.

This is the counterfactual query problem. The agent cannot estimate outcomes for actions absent from the data, so it cannot distinguish a real opportunity from a hallucinated one. When the dataset contains no transitions for a particular action in a particular state, any value assigned to that action is a guess dressed up as knowledge.

The practical rule: treat a value estimate as trustworthy only when the action sits inside the dataset's support—the region the behavior policy actually visited. Outside that support, the estimate is a hypothesis, not a measurement.

This is why offline RL methods constrain the policy to stay close to the data rather than chasing the highest raw value. The agent that optimizes without constraint will find the spot where the critic's extrapolation error is most positive—not the spot where the real environment is most rewarding.

How Offline RL Methods Cope: Staying Close to the Data

The solution families all share one instinct: limit how far the policy can wander from what the data supports.

Behavior cloning is the baseline. It treats the problem as supervised learning, matching the behavior policy's actions to its states. Training is stable and efficient, but the learned policy can at best match the behavior policy—it ignores reward information entirely and cannot improve on the data it imitates.

Policy regularization methods like TD3+BC add a behavior-cloning term to the policy objective. The agent still optimizes for return, but it pays a penalty for drifting away from the behavior policy's actions. The result is a policy that improves where the data supports improvement and stays quiet where it doesn't.

Value regularization methods like CQL take a different route. Instead of constraining the policy, they penalize the critic for assigning high values to out-of-distribution actions—actions unlike anything in the training data. The critic learns to be pessimistic about the unknown, which removes the phantom peaks that would otherwise attract the policy.

The shared tradeoff is fundamental: constraining toward the data limits how far the policy can improve beyond the behavior policy. Safety and improvement pull in opposite directions. Push too hard toward improvement and you risk extrapolation collapse. Push too hard toward safety and you've built an expensive behavior cloner.

Online vs Offline RL at a Glance

The two are not mutually exclusive. Many systems start offline on historical data, then fine-tune online to adapt to new conditions. The offline phase extracts what the logs can teach; the online phase corrects what the logs got wrong.

The practical decision is rarely a binary. Choose offline-first when interaction is unavailable or risky and the logged data is representative enough to support the decisions you need. Choose online when the environment is accessible and its feedback is trustworthy—a cheap simulator only helps if it models the real task faithfully. Choose a hybrid when offline data provides a safe starting policy but deployment can still collect corrective evidence. A simulator that diverges from reality will teach confident mistakes; a fixed dataset that misrepresents the deployment distribution will do the same.

Inspecting Support: From Concept to Diagnostic

The rule "trust a value only where the data has been" is only useful if you can check where the data has been. In practice, support is not a label printed on the dataset—it is something you approximate by inspection.

Start with the behavior policy. What generated the logs, and how diverse was its behavior? A dataset from a single deterministic controller has narrow support by construction, no matter how many transitions it contains.

Then compare the policy's selected actions against the logged actions at similar states. For discrete actions, check whether the chosen action appears in the data for that state or nearby states. For continuous actions, look at the density of logged actions in the neighborhood of the proposed action—nearest-neighbor distance is a crude but useful proxy. A selected action that sits far from any logged action at a similar state is an uncertainty flag, not a proven failure.

Treat these checks as diagnostics, not ground-truth evaluation. They tell you where the critic is guessing, not whether the guess is wrong. But that distinction is exactly the point: an estimate outside the support is a hypothesis, and a hypothesis needs correction before it can be trusted.

The Mental Model That Keeps You Honest

The durable rule: in offline reinforcement learning, trust a value only where the data has been. Treat every estimate outside the dataset's support as a guess, not a fact.

The dataset is the environment's only voice. If it never spoke about an action, the agent has no right to be confident about it. A high value on an unsupported action is not a discovery—it's an artifact of generalization without correction.

Before trusting any offline result, ask three questions. What was the behavior policy, and how competent was it? How diverse was its coverage across states and actions—not just globally, but conditional on the states your policy actually visits? Does the high-value action the policy selected sit inside the logged support, or is the critic extrapolating across a gap? If you cannot answer all three, you are not evaluating an agent—you are evaluating a hallucination.

The next step is turning these questions into a systematic evaluation procedure: measuring learning curves across seeds, comparing against baselines, and isolating whether a failure comes from the algorithm or from the data's silent regions.