Safe Policy Improvement From Logged Data: Staying Near What the Dataset Supports

A higher estimated value is not proof of improvement. It is a hypothesis about actions the dataset may never have supported.

Why a Higher Estimated Value Is Not Proof of Improvement

Offline reinforcement learning has a failure pattern that repeats across methods and datasets: a policy that selects out-of-distribution actions receives inflated value estimates that never materialize in deployment. The mechanism is extrapolation error—when the value function is asked to score state-action pairs that never appeared in the logged data, it extrapolates from whatever it has seen, and that extrapolation is frequently optimistic.

The practical consequence is uncomfortable. You can run an offline RL algorithm, watch the estimated return climb, deploy the resulting policy, and watch real performance fall below the behavior policy that collected your data in the first place. The estimate was not lying exactly. It was guessing in a region where it had no evidence, and the guess happened to be flattering.

This is why safe policy improvement offline reinforcement learning reframes the problem. The goal is not to maximize estimated return. The goal is to avoid regression below the behavior policy that generated the logged data. That is a different standard, and it demands different decision criteria.

Three criteria separate a defensible improvement from a lucky estimate:

1. Dataset coverage — are the actions the new policy favors actually represented in the data?
2. Uncertainty — did the method discount values where evidence is thin?
3. Policy deviation — was the distance the policy moved from the behavior policy bounded?

Keep those three in mind. Everything else in this article is detail.

The Baseline Guarantee: Improvement Measured Against the Behavior Policy

Safe policy improvement is defined relative to the behavior policy that generated the logged data—not against an unknown optimal policy, and not against the best policy you can imagine.

That distinction matters more than it first appears. Ordinary offline RL asks, "What is the best policy this data can support?" Safe improvement asks a narrower question: "Can I avoid regression below the baseline that collected this data?" The first question invites risk. The second one starts from a floor.

The practical consequence is that conservative methods often accept a smaller gain in exchange for a defensible lower bound on performance. If the behavior policy achieves a return of 80, a safe improvement method might deliver 83 with a guarantee that it will not fall below 78. An unconstrained method might report an estimated 95—and then deliver 60 when the extrapolation error catches up with it.

One terminology note before we go further. "Safe" here means no regression relative to the behavior policy. It does not mean safety in the constraint-satisfaction sense—no collisions, no cost-limit violations, no hard operational boundaries. Those are different guarantees. A method can be safe in the no-regression sense while still proposing actions that violate a cost constraint, and a method can satisfy cost constraints while regressing below the behavior policy. Keep the two meanings separate, because the literature uses both.

Criterion One: Dataset Coverage as the Boundary of Trust

Dataset coverage is the set of state-action pairs the logged data actually visits. It is the hard boundary for any offline claim. No amount of algorithmic sophistication creates evidence that was never collected.

When you evaluate an improvement step, the first question is not "What is the estimated value of this action?" It is "Does the dataset contain enough evidence to estimate the consequences of this action at all?"

Coverage has two distinct senses, and conflating them causes real mistakes:

• Support coverage asks which actions appear at all. If the dataset contains any instance of taking action a in state s, then a is in the support of the data for that state.
• Density coverage asks how often each action appears. An action can be in the support—present somewhere in the dataset—but appear so rarely that the value estimate built from those few samples is noise.

The distinction matters because support and density justify different levels of trust. An action that appears once in a million transitions is supported in the technical sense, but the value estimate attached to it is built from a single observation. That is not evidence; it is an anecdote.

The diagnostic question for any proposed improvement: for the actions the improved policy would take, does the dataset contain enough evidence to estimate their consequences? If the answer is no, the improvement is not defensible—regardless of what the value estimates claim.

This support-versus-density distinction also previews the two constraint families we will discuss shortly. Some methods constrain the policy to stay within the support of the behavior policy. Others push the policy toward the behavior policy's full action distribution. The choice between them is not a minor implementation detail. It determines how much improvement is even possible.

Criterion Two: Uncertainty as the Reason to Be Conservative

Value estimates carry uncertainty, and that uncertainty grows as the policy moves away from well-covered actions. A point estimate—"this action has value 95"—hides the real situation, which is usually "this action has an estimated value of 95, but the estimate is built from very little evidence and could easily be wrong by a wide margin."

Defensible methods respond to this with a pessimistic correction. When evidence is thin, they discount or penalize the estimated value rather than trusting it. The mechanism varies—some methods lower the Q-value for actions that fall outside the data distribution, others add a penalty term that grows with uncertainty, others simply refuse to update values for unsupported actions—but the principle is consistent: uncertainty is treated as a reason to trust less, not a reason to hope more.

There are two failure modes here, and they sit at opposite ends of the same spectrum.

The first is optimistic extrapolation: trusting an overestimated value because the method had no mechanism for discounting unsupported actions. This is the classic offline RL failure, and it produces policies that look excellent in evaluation and collapse in deployment.

The second failure mode is less discussed but equally real: being so pessimistic that the policy never improves beyond the behavior policy. If every action that deviates from the data distribution is heavily penalized, the method will find that the safest choice is always to imitate the behavior policy. The guarantee holds—no regression—but the improvement is zero.

The decision rule that threads between these failure modes: the size of the allowed improvement step should shrink as uncertainty about the target actions grows. When uncertainty is low, you can afford to move. When uncertainty is high, you should stay close to what the data supports.

Criterion Three: Policy Deviation as the Levers You Control

Dataset coverage is a property of your data. Uncertainty is a property of your estimates. Policy deviation is the design lever you actually control: the method chooses how much the improved policy may differ from the behavior policy.

Constraint families differ in what they measure and therefore in what they permit.

Density-based constraints push the new policy toward the behavior policy's action distribution. They measure the distance between the two distributions—often with KL divergence or a similar metric—and cap how far the new policy can drift. These constraints are straightforward to implement and easy to reason about. They also have a serious weakness: they can block improvement toward a better action that is merely underrepresented rather than absent.

Consider a dataset where the behavior policy chose action a 95% of the time and action b 5% of the time. If action b is actually much better, a density-based constraint will resist shifting probability mass toward it, because doing so increases the distance between the new policy and the behavior policy's distribution. The constraint protects the policy from unsupported actions, but it also protects the status quo from evidence.

Support-based constraints take a different approach. They require only that the new policy stay within the set of actions the data covers—not that it match the frequency with which those actions were taken. Under a support constraint, the policy is free to shift probability mass from action a to action b, as long as action b appears somewhere in the dataset.

The tradeoff between the two families is the central design decision in conservative policy improvement:

• Tighter constraints are safer but may forfeit real gains by blocking movement toward underrepresented actions.
• Looser constraints capture more improvement but raise the risk of selecting actions whose support in the data is too thin to support reliable estimates.

There is no universally correct choice. There is only the question of what you can defend given your data, your uncertainty estimates, and your tolerance for regression risk.

Reading a Method Through These Three Criteria

When you encounter an offline RL method—in a paper, a library, or a colleague's experiment—run it through the three criteria. The mental checklist is short:

1. Coverage: Does the method know which actions the dataset supports? Does it distinguish between actions that appear at all and actions that appear often enough to support reliable estimates?
2. Uncertainty: Does it discount or penalize values where evidence is thin? Or does it treat every estimated Q-value as equally trustworthy?
3. Deviation: Does it bound how far the policy can move from the behavior policy? And does that bound use density (matching the action distribution) or support (staying within the covered action set)?

A defensible method typically combines a pessimistic Q-update with a policy constraint. The pessimistic update ensures that actions with thin support receive discounted values, so the policy improvement step is not seduced by optimistic extrapolation. The policy constraint ensures that even if the value estimates are wrong in some region, the policy cannot move far enough to suffer catastrophic regression. The two mechanisms work together: pessimism makes the estimates honest, and the constraint limits the damage if they are still wrong.

One common mistake deserves a direct warning. A single high evaluation score on logged data is not proof of safety. Off-policy evaluation can repeat the same extrapolation error that corrupted the training values—if the evaluation method relies on the same value estimates, it will confidently endorse the same unsupported actions. Treat evaluation scores as evidence to inspect, not as certificates of safety.

There is also an open question you should keep in mind. Many guarantees in the safe policy improvement literature assume no approximation error and no sampling error. Real implementations always have both. The guarantees weaken accordingly. A method that provably improves the policy in the idealized setting may only approximately improve it in practice—which is exactly why the three criteria matter more than the proof.

When Safe Improvement Is the Wrong Tool

Conservative improvement is a tool with a specific job. It is not always the right tool.

When not to use it: If your dataset has broad, dense coverage and your task tolerates exploration, ordinary offline RL or online fine-tuning may capture more value. The conservative machinery exists to protect against extrapolation error. If your data already covers the relevant state-action space densely, that protection is less necessary—and the constraints will cost you improvement you could have safely captured.

When it is insufficient: If your goal is to satisfy hard safety constraints—cost limits, collision avoidance, operational boundaries—no-regression guarantees alone do not enforce them. A policy that never regresses below the behavior policy can still violate a cost constraint if the behavior policy itself was unsafe. Safe improvement and safe constraint satisfaction are different problems, and you need to know which one you are solving.

When it is overkill: If you only need a research baseline or a diagnostic, the overhead of conservative constraints may obscure the mechanism you are studying. When the goal is understanding how an algorithm behaves, you often want the unconstrained version so you can see the failure modes directly.

The decision rule that ties the three criteria together: before accepting any offline improvement, ask whether the actions it favors are covered by the dataset, whether the value estimates carry uncertainty that was discounted, and whether the policy deviation was bounded. If all three answers are yes, the improvement is defensible. If any answer is no, you are not doing safe policy improvement—you are hoping.

The natural next step is evaluating the improved policy with off-policy evaluation methods, which estimate a target policy's performance from logged experience without trusting the training-time value estimates. That evaluation is where the three criteria meet their test: a defensible improvement should survive scrutiny from an independent estimator, and an indefensible one should not.