Dataset Support and Coverage in Offline RL: Can the Logs Answer Your Question?

You train an offline agent on a fixed dataset. The final score looks strong. The learning curves climb. You feel the quiet satisfaction of a result that seems ready to report.

Then someone asks the uncomfortable question: What exactly did the logs prove?

Here is the trap. A high offline score does not mean the logs proved your policy works. It means the logs proved your policy works where the data happened to look. Those are different claims, and confusing them is how offline RL results quietly fall apart.

The real question is not whether your dataset is "good." It is whether your dataset can answer the specific question you are asking.

The Question the Logs Can and Cannot Answer

Dataset quality is never absolute. A dataset is not good or bad in the abstract—it is adequate or inadequate for a particular question.

The same logs can support one target policy and silently betray another. Imagine a dataset collected by a cautious robot that never approached the edge of a table. That dataset could support a policy that keeps the robot near the center. It cannot support a policy that leans over the edge, no matter how many transitions it contains.

Before you evaluate anything, name the question you are asking. In offline RL, you are usually asking one of two things:

• Evaluation: Does this target policy perform well?
• Improvement: Can I learn a policy that beats the behavior that collected the data?

These questions share a hidden anchor criterion: the target policy must stay where the logs have evidence. But they demand different kinds of evidence, and treating them as identical is where confident conclusions start to drift.

Evaluation vs. Improvement: Two Different Demands

Evaluation asks the logs to describe a policy you already have. Improvement asks the logs to justify choosing a policy you do not have yet. Those are different burdens.

What you are doing	What the logs must show	Where it gets fragile
Evaluation	The target policy's actions and the states it visits are observed often enough to estimate its outcomes	The target policy visits states or takes actions the behavior policy rarely reached
Improvement	Candidate policies that beat the behavior policy are represented, and their advantages are estimable from the data	The "better" policy depends on actions whose consequences the logs never demonstrated

Evaluation is a report. Improvement is a bet. A report needs enough observations of the policy you are describing. A bet needs enough evidence that the alternative you are choosing is genuinely better—not just higher-scoring inside a lucky region of the estimate.

This is why improvement is the more demanding claim. A policy can pass an evaluation check and still fail as an improvement target, because improvement often requires comparing many candidate policies across a sequence of decisions. Each candidate needs support, not just the one you finally report.

Support: What the Logs Actually Contain

Dataset support is the set of state-action pairs the logged data actually visited. It is not the state space the environment permits. It is the much smaller region the behavior policy happened to occupy.

Think of it as the difference between a map of a city and the streets you have actually driven. The map shows every road the environment allows. The support is the handful of routes you have evidence about. A dataset collected only near a start position cannot support conclusions about behavior far from it, no matter how many episodes it contains.

Now sharpen that image with three levels of evidence:

• Possible. The environment allows this state-action pair. The map shows the road.
• Observed. The logs contain at least one visit to this pair. You have driven the road once.
• Reliably estimated. The logs contain enough visits to distinguish a real pattern from noise. You have driven the road enough times to know what happens.

Most coverage discussions collapse these three levels into one. That collapse is where overconfident conclusions are born. A single rare visit to a state-action pair is not the same as reliable support. If a robot tried one risky action once and never repeated it, the logs contain that pair—but they contain almost no information about its consequences. One sample cannot tell you whether that action leads to reward or disaster.

The practical image to hold: the logged region is a shaded area in state-action space. A target policy that stays inside the shaded area is asking the logs to speak where they have evidence. A target policy that exits the shaded area is asking the logs to invent an answer.

The Behavior Policy Is the Ceiling You Did Not Choose

Every offline dataset has a hidden author: the behavior policy that collected the data. It might be a random explorer, a noisy expert, a replay buffer from online training, or a mixture of sources. Whatever it was, it silently sets the boundaries of what is learnable.

Here is the nuance that matters: a target policy does not need to resemble the behavior policy to be supported. It needs to select actions whose consequences the logs can estimate in the states the target policy actually visits. A policy can differ from the behavior policy and still be well-supported, as long as the relevant alternatives were tried often enough to judge.

The reverse is also true. A policy that looks similar to the behavior policy can still fail if it makes a rare decision at a critical state—one unlucky divergence at the wrong moment can push it into territory the logs never mapped.

The practical signal I want you to remember: if your learned policy rarely resembles the behavior policy, treat every high score with suspicion. The algorithm may have found a policy that looks excellent on paper precisely because it wandered where the logs could not correct it.

Coverage Is Not Uniform: Density, Reachability, and the Long Tail

Coverage is a distributional property, not a binary one. A dataset can cover a state well and an action poorly. It can cover early states densely and later states sparsely. It can look comprehensive at a glance and contain holes exactly where your target policy needs to go.

Two properties matter beyond raw presence:

Density. How many samples exist in a region? Dense regions support reliable estimates. Thin regions produce noisy ones.

Reachability. A state may appear in the logs only because the behavior policy stumbled into it by accident—not because the target policy can reliably get there. If the behavior policy reached a dangerous state once through luck, the logs contain that state, but they do not contain the path that reliably leads to it.

The long tail of rare transitions is where offline RL quietly fails. The algorithm can memorize the common path and hallucinate the rest. It learns to navigate the dense center of the data distribution, then invents confident predictions about the sparse edges.

This is why many practical guarantees in offline RL only hold for policies that stay within the covered region. The theoretical term is partial coverage: the data only covers the state-action distribution of some comparator policy, not the entire environment. In plain language: the logs can support a policy whose relevant decisions are represented, and they cannot justify gains that depend on actions or transitions the data never demonstrated.

That last distinction is worth sitting with. Offline improvement is not imitation. A learned policy can beat the average behavior of the data collector when the better alternatives are present in the logs and their consequences are estimable. What it cannot do is justify an advantage that rests on unsupported territory—actions whose outcomes the logs never observed.

Why the Target Policy Drifts Out of Distribution

You might wonder: if leaving the data is dangerous, why do learned policies keep doing it?

Because optimization pushes them there.

The objective in offline RL rewards actions that look best. And the actions that look best are often the least-visited ones—because their estimates are noisy, and noise tends to be optimistic. An action tried once with a lucky reward looks like a goldmine. The algorithm reaches for it. The estimate was a mirage, but the policy has already committed.

This is the core tension of offline RL: the objective rewards leaving the data, while reliability requires staying inside it. The algorithm is not buggy. It is doing exactly what you asked—maximizing estimated return—and the estimates are misleading it where the data is thin.

A policy that achieves a high offline score by exploiting unsupported actions is not a good policy. It is a policy that found a hole in the estimate. The score measures the hole, not the policy.

Reading a Dataset Before You Trust It

Before you run an algorithm, spend time reading the dataset. You do not need sophisticated tooling—you need the right questions.

Check the behavior policy. What collected this data? Random exploration, a trained expert, a noisy intermediate policy, a replay buffer? Each leaves a different fingerprint. A dataset from a random policy covers broadly but shallowly. A dataset from an expert covers narrowly but deeply. Know which one you have.

Look at state-action coverage conditionally. Global action diversity is weak evidence. A dataset can show broad action variety overall while having poor coverage at the specific states your target policy visits. The operational test is conditional: for the states and time points along your target policy's likely trajectory, which actions were actually tried, and how often?

Trace where your target policy would go. Walk through the states it would visit and the actions it would select. Does it stay in regions with dense evidence, or does it cross into sparse territory? The shaded-region image becomes concrete when you trace an actual trajectory through your data.

Use cheap statistical signals. Episode lengths, reward spread, and action diversity tell you a lot. Short episodes with narrow reward ranges suggest limited behavior. Wide action diversity suggests exploration. A dataset where every episode looks identical may not contain enough variation to support improvement.

Common mistake: Judging coverage from a global visualization or a single aggregate number. A dataset can look well-covered in the aggregate while hiding holes exactly where your target policy needs evidence. Always inspect coverage conditional on the states your policy actually visits.

When in doubt, prefer a conservative claim. Report what the logs support rather than what the algorithm optimistically predicts. A modest claim that survives scrutiny beats an impressive claim that collapses when someone checks the coverage.

When Coverage Thinking Applies (and When It Does Not)

Coverage reasoning is essential for offline evaluation and improvement, but it is not the only constraint, and it does not apply the same way online.

Use coverage reasoning when: you are judging whether an offline dataset can support a target policy, comparing two datasets for a task, or deciding whether an offline result is trustworthy enough to report.

Do not over-apply it when: the real bottleneck is reward design, representation quality, or a buggy environment. Coverage analysis will not fix a reward that rewards the wrong behavior. It will not fix a state representation that hides what the agent needs to know. It tells you where the logs can speak—not what the environment rewards or what a good policy looks like.

Online RL relaxes the constraint. An online agent can explore to fill gaps. It can try an action, observe the consequence, and update its belief. Offline RL cannot. The dataset is frozen. This is why coverage becomes the binding limit offline: the agent can never correct a missing region by visiting it.

Keep the mental model clean: coverage tells you where the logs can speak. It does not tell you what the logs say.

The Decision Rule

Before you trust any offline result, run this sequence:

1. Name the target policy. What behavior are you actually evaluating or trying to learn?
2. Locate it relative to the logged evidence. Does it stay where the logs have enough density to estimate its outcomes, or does it reach into sparse territory?
3. State plainly what the logs can and cannot support. Write the claim in one sentence. If the sentence requires the logs to speak where they have no evidence, the claim is not ready.

The next step is concrete: inspect a real dataset's coverage before you run an algorithm. Look at its behavior policy, its state-action distribution conditional on the states your target policy visits, and its gaps. The logs are not a neutral resource. They are a bounded witness—and your job is to know exactly what they witnessed before you let them testify.