Off-Policy Evaluation in Reinforcement Learning: Estimating a Policy From Logged Experience

You have a new policy that looks promising. The next step feels obvious: deploy it, watch it interact with the real environment, and measure what happens. But deployment is expensive, risky, slow, or genuinely unsafe. What if you could estimate how well the new policy would perform before letting it touch the live system?

That is the promise of off-policy evaluation in reinforcement learning: judging a policy you have never run, using only experience collected by a different policy. The catch is that the answer is only as trustworthy as the assumptions underneath it. Get those assumptions wrong, and you will ship a confident estimate that quietly measures the wrong thing.

Why You Cannot Just Replay the Logs

Imagine you have a dataset of logged episodes. Each episode records states, actions, rewards, and the next states that followed. Your instinct is simple: replay the logs, average the returns, and call that the value of your new policy.

That instinct is wrong in a specific, diagnosable way.

The logs were not produced by your new policy. They were produced by whatever policy was running when the data was collected—perhaps an older model, a heuristic, or a human operator. That data-collecting policy is called the behavior policy. The policy you want to judge is the target policy. When you naively average the logged returns, you are measuring the behavior policy's performance, not the target policy's. The actions in the logs were chosen by the behavior policy, and the rewards that followed were consequences of those choices.

This mismatch between behavior policy and target policy is the entire problem of off-policy evaluation. The goal is not to replay the logs. The goal is to correct them so they speak for a policy that never collected them.

Think of it this way: the logs are a witness who only saw one version of events. You want to know what would have happened under a different script. The witness cannot re-enact the scene, but the evidence they gathered might still tell you something—if you know how to adjust for what they actually saw.

The Coverage Assumption: What the Logs Must Contain

Before any estimator can work, one precondition must hold: the behavior policy must have explored the states and actions the target policy would choose.

This is the coverage assumption. Every action the target policy might take must appear with nonzero probability in the behavior policy's data. If the target policy picks an action the logs never recorded, no estimator can honestly recover its value. The data simply does not contain the evidence needed.

Coverage failure is not a subtle statistical edge case. It is a hard boundary. If your target policy favors an action that the behavior policy chose only 0.1% of the time, you will need enormous datasets to estimate anything reliably. If the behavior policy never chose it at all, the estimate is not noisy—it is impossible.

This is the same boundary that drives extrapolation error in offline reinforcement learning. When you try to learn from a fixed dataset, unsupported actions create unreliable value targets. When you try to evaluate from logged data, unsupported actions make the target policy's value unidentified: the logs contain no evidence about what would happen if the target policy took them. The difference is the goal. Offline RL learns a policy despite the boundary, while off-policy evaluation tries to judge one honestly within it. A partially unsupported dataset may still support a partial judgment—but not a defensible full-policy estimate.

Before trusting any estimate, run a practical check. Inspect the action frequencies in your logs. Ask: for every state the target policy might visit, did the behavior policy take the actions the target policy would favor? If the answer is no, stop. No estimator can rescue missing coverage.

Importance Sampling: Reweighting Each Step

When coverage holds, you can correct the mismatch between policies using importance sampling. The idea is elegant: weight each logged action by how much more or less likely the target policy would have chosen it compared to the behavior policy.

For a single action in a single state, the weight is a ratio:

target policy probability of the action
------------------------------------------
behavior policy probability of the action

If the target policy would have chosen that action twice as often as the behavior policy did, the ratio is 2. If it would have chosen it half as often, the ratio is 0.5. Multiply the observed reward by this ratio, and you have corrected for the fact that the wrong policy chose the action.

But an episode is not a single action. It is a chain of decisions, where each choice changes the state the next choice sees. To correct for the whole trajectory, you multiply the ratios across every step. The product of per-step ratios becomes the trajectory's importance weight.

Why the product? Because the probability of seeing an entire trajectory under the target policy, relative to the behavior policy, is the product of the per-step action probabilities. Each step's correction compounds. If the target policy diverges from the behavior policy at several points in a trajectory, the product grows or shrinks accordingly.

This estimator is unbiased: averaged over enough data, it converges to the true value of the target policy. But unbiasedness is not the same as usefulness.

A Worked Example: Watching the Correction Move

Let's make this concrete with a short three-step episode. Suppose the behavior policy chooses uniformly among three actions, so each action has probability 1/3. Your target policy is different: it strongly prefers action A.

Step	Logged action	Reward	Behavior prob.	Target prob.	Step ratio	Cumulative ratio
1	A	+1	1/3	0.8	2.4	2.4
2	B	0	1/3	0.1	0.3	0.72
3	A	+2	1/3	0.8	2.4	1.728

The episode's total return is 3. The trajectory importance weight is the product of the step ratios: 2.4 × 0.3 × 2.4 = 1.728. The trajectory-wise importance-sampling estimate of the target policy's value is that product times the total return: 1.728 × 3 = 5.184.

Now watch what happens at step 2. The behavior policy chose B, but the target policy rarely would have. The step ratio is 0.3, which shrinks the cumulative weight. That is the correction working: this episode contains a decision the target policy would probably not make, so the episode should count less.

There is a second way to apply these weights. Instead of weighting the whole return by the cumulative ratio, you can weight each reward by the ratio accumulated up to that step. This is per-decision importance sampling. It produces a different estimate with different variance properties. For this article, the key distinction is simple: trajectory-wise weighting multiplies the full return by one cumulative ratio, while per-decision weighting applies step-by-step ratios to individual rewards. Both correct for the policy mismatch; they just distribute the correction differently across time.

Now consider what happens when the target policy diverges sharply. Suppose the behavior policy chooses uniformly among ten actions, and your target policy almost always picks action 7. When the logs happen to contain action 7, the importance ratio is roughly 10 divided by 0.1—about 100. Those rare episodes dominate the estimate. A few lucky trajectories where the behavior policy stumbled into action 7 will swamp everything else. The estimate is correct on average, but any single estimate can be wildly unstable.

This is the core tension of off-policy evaluation: the correction that makes the estimate unbiased is the same mechanism that makes it noisy.

Why Variance Explodes and What to Do About It

The variance problem is not a nuisance. It is the dominant practical failure mode of plain importance sampling.

When the target policy favors actions the behavior policy rarely took, the importance ratios grow large. A handful of episodes with high ratios can dwarf thousands of ordinary ones. Your estimate becomes a lottery: either the logs happened to contain a few high-weight trajectories, and the estimate is inflated, or they did not, and the estimate is deflated. The average over many datasets is correct, but you only have one dataset.

The standard stabilization is self-normalized importance sampling. Instead of averaging the raw weighted returns, you divide by the sum of the importance weights themselves. This bounds the estimator and reduces variance dramatically. The tradeoff is a small bias: the estimator is no longer exactly unbiased, but it is far more stable. In practice, that stability is usually worth more than the theoretical purity of unbiasedness.

For longer horizons, even self-normalized importance sampling struggles. The product of ratios across many steps can still explode. Researchers have developed alternatives that lean on learned models of the environment or value functions. Doubly robust estimators combine importance sampling with a learned value model: the model provides a baseline prediction, and the importance weights correct only the residual error. If the model is good, variance drops. If the model is wrong, the importance sampling component still provides a safety net.

Before reaching for these advanced methods, check whether you can trust the learned model they depend on. A value model trained on the same logs carries its own errors, especially in regions the behavior policy visited rarely. Doubly robust estimation is not an automatic upgrade; it is a bet that your model's predictions are informative enough to reduce variance without adding bias.

Choosing an Estimator: A Practical Decision Rule

You now have a family of estimators, each with a different balance of bias and variance. How do you choose?

The axis that matters most is how far the target policy strays from the behavior policy. That distance determines how large the importance ratios can grow, which determines how much variance you must absorb.

Situation	Estimator choice	Why
Target policy close to behavior policy	Plain importance sampling	Ratios stay near 1, variance is manageable, unbiasedness is preserved
Target policy diverges moderately	Self-normalized importance sampling	Bounded weights tame variance at the cost of small bias
Long horizons or large divergence	Model-based or doubly robust	Learned value model reduces reliance on raw ratios
Coverage is thin or uncertain	No estimator can save you	Check the data before choosing a method

Whatever you choose, resist the urge to overclaim. An estimate with tight error bars on paper can still be wrong if coverage is thin or if your model of the behavior policy is inaccurate. The guarantees behind these estimators assume you know the behavior policy's action probabilities. In practice, you often have to estimate them from the same logs you are evaluating—and an inaccurate behavior policy model quietly corrupts every ratio you compute.

What is known versus open matters here. The theory of off-policy evaluation is well developed for the case where the behavior policy is known or well estimated. When it is not, the guarantees weaken, and the practical reliability of your estimate depends on judgment rather than mathematics.

Common Mistakes When Evaluating From Logged Data

After working through the workflow, four mistakes appear again and again. Run these as diagnostic questions against your own setup.

Mistake 1: Treating the average logged return as the target policy's value. This is the replay trap from the opening. You are measuring the behavior policy, not the target policy. Ask: did I correct for the fact that a different policy chose these actions?

Mistake 2: Ignoring coverage. An estimate built on actions the target policy would rarely or never take is not an estimate—it is a guess wearing a confidence interval. Ask: does the logged data actually contain the decisions my target policy would make?

Mistake 3: Using an inaccurate model of the behavior policy. If you estimated the behavior policy's probabilities from the logs, those estimates carry their own error. Ask: how sure am I about the behavior policy's action probabilities, and what happens to my ratios if I am wrong?

Mistake 4: Reporting a single point estimate. A number without a sense of variance or confidence is nearly useless for a deployment decision. Ask: how much would this estimate move if I resampled the data or used a different estimator?

The through-line is humility. Off-policy evaluation is a tool for reducing risk before deployment, not a crystal ball that eliminates it. The estimate earns trust the same way any experimental result does: by making its assumptions visible and its uncertainty legible.

A Small Experiment to Test Your Estimate

Before you trust any off-policy estimate, run a compact logged-data experiment. The goal is to make the estimate's fragility visible.

First, verify coverage: confirm that every action your target policy might favor appears in the logs with reasonable frequency. Second, confirm you know the behavior policy: if you estimated its probabilities from the logs, treat that estimation error as part of your uncertainty.

Then run the experiment. Resample your logged episodes with replacement to create several bootstrap datasets. Compute your chosen estimator on each resample and record the spread of estimates. A tight spread suggests your estimate is stable. A wide spread means a few high-weight episodes are driving the result—a sign of thin coverage or strong policy divergence.

Next, compare at least two estimators. Run plain importance sampling and self-normalized importance sampling on the same data. If they agree closely, you have some confidence the estimate is not an artifact of the estimator's mechanics. If they disagree sharply, the largest importance weights are probably dominating, and you should treat the estimate with suspicion.

Finally, inspect the largest weights directly. If a handful of episodes carry most of the total weight, your effective sample size is small, no matter how many episodes you logged. That visible spread will teach you more about the reliability of your evaluation than any theoretical guarantee.

The same coverage boundary that constrains evaluation becomes a learning constraint in offline RL. Once you can judge a policy honestly from logs, you have the foundation to ask the harder question: how do you learn a good policy when you can never interact with the environment? That is where the coverage assumption stops being a statistical nicety and becomes the central design problem.