Test Reinforcement Learning Policy Generalization Across Held-Out Environments

Your policy scores well on the training environment. You change one parameter, run it again, and the score falls off a cliff. Is that a transfer failure, or did you just get an unlucky seed?

You cannot answer that question with a single run. You need a split, a seed sweep, and a gap you can compare against noise. This is a measurement exercise, not an algorithm exercise. By the end, you will have a table that tells you whether your policy actually fails to generalize or whether your measurement is simply too noisy to say anything at all.

Why One Held-Out Run Proves Nothing

The default mistake looks like this: train once, evaluate once on a slightly different environment, see the score drop, and conclude "my policy does not generalize." That conclusion hides two completely different failure sources inside one number.

Transfer failure means the policy latched onto training-specific details—a particular starting position, a specific friction value, a narrow goal placement—and those details do not survive the change. The policy learned the environment, not the task.

Training variability means the seed produced a weak policy to begin with. Deep RL is brittle with respect to random seed. Two runs of the same algorithm on the same environment can produce learning curves that look like they came from different algorithms entirely. If your single run happened to be a weak one, its held-out score tells you nothing about generalization.

A held-out score is only interpretable relative to the spread of scores the same algorithm produces across seeds on the training environment. That spread is your noise floor. If you already know that repeated episodes and seeds matter for evaluation, good. This article turns that principle into a concrete split-and-compare procedure you can run today.

Define the Split Before You Train

Pick one environment and one algorithm you can already run. Do not change the algorithm during this exercise. The algorithm is not the variable you are testing.

Now choose a variation axis—a single dimension along which the environment can change in a controlled, named way. Good candidates:

• Initial-state ranges (starting position, initial velocity, initial angle)
• A dynamics parameter (friction, motor power, gravity, mass)
• Obstacle or goal placement
• A small set of task variants (different goal locations, different object configurations)

Keep the training distribution narrow. Train on a restricted range of that axis—say, friction between 0.8 and 1.0. Then hold out three to five variations entirely: friction at 0.5, 0.6, 0.7, and maybe one at 1.2. These are genuinely unseen.

Decide the variation axis before training, not after seeing results. If you look at the results first and then choose which variations to hold out, you have tuned the held-out set into the training set. The split stops being a test.

Record the exact configuration of each variation. Write it down. If you cannot reproduce the split, you cannot attribute a failure to a named change.

Common mistake: Choosing held-out variations that are secretly in-distribution. If your training range is friction 0.8–1.0 and your "held-out" variation is friction 0.85, you have not held anything out. The variation must be outside the training range.

Run the Seed Sweep and Log the Right Numbers

Fix everything except the seed. The seed is the only variable in this first pass. Run at least three seeds. Ten is the point where the spread estimate becomes trustworthy rather than anecdotal. If you can only afford three, run three—but know that your noise floor estimate is rough.

For each seed, evaluate the final policy on the training split and on every held-out variation. Use multiple episodes per evaluation. Report the mean and spread, not a single episode.

Here is a runnable scaffold. It assumes a small interface you almost certainly already have in some form: a make_env(config, seed) factory, a train(env, seed) function that returns a policy, and an evaluate(policy, env, episodes) function that returns a mean return. The point is not the exact function names—it is the loop structure and the output schema.

import statistics

SEEDS = [0, 1, 2, 3, 4]
EPISODES = 20

train_config = {"friction": (0.8, 1.0)}
held_out = {
    "A": {"friction": 0.5},
    "B": {"friction": 0.6},
    "C": {"friction": 0.7},
}

rows = []
for seed in SEEDS:
    train_env = make_env(train_config, seed=seed)
    policy = train(train_env, seed=seed)

    train_score = evaluate(policy, train_env, episodes=EPISODES)

    held_scores = {}
    for name, cfg in held_out.items():
        env = make_env(cfg, seed=seed)
        held_scores[name] = evaluate(policy, env, episodes=EPISODES)

    mean_held = statistics.mean(held_scores.values())
    rows.append({
        "seed": seed,
        "train": train_score,
        "held": held_scores,
        "gap": train_score - mean_held,
    })

for r in rows:
    print(r["seed"], r["train"], r["held"], round(r["gap"], 1))

The output is a flat table. Each row is one seed, with the training score, the held-out scores, and the paired gap for that seed:

Look at the training-split scores first. They already disagree across seeds—850, 920, 780, 910, 840. That disagreement is your noise floor. The algorithm, on the environment it was trained on, produces scores that vary by roughly 140 points depending on seed.

Now you can read the held-out scores against that baseline.

Read the Gap Against the Noise

The gap column is the number that matters. It is the per-seed difference between what the policy earned on the training split and what it earned on the held-out variations. Because each gap is computed within a single seed, it already controls for that seed's training quality.

The training-score spread—about 140 points here—tells you how much the algorithm's final performance moves around for reasons that have nothing to do with transfer. It is context, not a hard cutoff. A gap larger than that spread is a signal worth taking seriously. A gap smaller than it is not evidence of anything, because ordinary seed variation could produce it.

Read the table row by row:

• Seed 1 and Seed 3: Gaps of 105 and 110. These are within the seed-to-seed spread. You have not demonstrated a transfer failure. You have demonstrated that your measurement is too noisy to tell.
• Seed 2: Gap of 325. This is clearly larger than the noise floor. This seed shows evidence of a real generalization gap.
• Seed 0 and Seed 4: Gaps of 180 and 195. These are borderline—larger than the spread but not dramatically so.

The mixed case is the most informative: some seeds transfer, some collapse. That pattern points at instability interacting with the variation, not a uniform policy failure. The policy is not consistently bad at transfer; it is inconsistently good at training.

Here is the decision table:

Tip: If you see the seed-dependent pattern, do not immediately widen the training distribution. First, check whether the collapsing seeds also underperform on the training split. If they do, the problem is training instability, not transfer.

Failure Modes That Fake a Generalization Result

Before you blame the policy, debug the measurement. These are the concrete ways this experiment misleads.

Too few seeds. Two runs can differ enough to look like different algorithms. A gap computed from one or two seeds is not evidence. Run at least three, preferably ten.

Held-out variations that are secretly in-distribution. If the variation falls inside the training range, you are not testing transfer. Check the actual parameter values.

Held-out variations that are too far out. If the variation is so extreme that no policy could handle it, failure says nothing about transfer. The variation should be outside the training range but still within the realm of the task.

Single-episode evaluation. One episode mixes episode luck into the generalization gap. Use multiple episodes and report the mean.

Comparing mismatched checkpoints. If you evaluate the final checkpoint from one seed against the best checkpoint from another, you confound checkpoint selection with transfer. Use the same checkpoint criterion for every seed.

Accidental training on held-out variations. Shared random state, environment reuse, or a wrapper that samples from the full variation set can leak held-out variations into training. Verify that your training loop only sees the training configuration.

Warning: The most common version of this bug is a wrapper that samples from the full variation set during training "for robustness" and then evaluates on a subset of that same set. Check your environment wrapper before you trust any gap.

One Follow-Up Experiment: Widen the Training Distribution

You have a table. You have a gap—or you have noise. Now test the mechanism behind the gap.

Keep the held-out set fixed. Change only the training distribution: train on a wider range of the same variation axis. If your original training range was friction 0.8–1.0, widen it to 0.6–1.1 and keep the held-out set at 0.5 and 1.2. The held-out variations remain outside the training range, but the training distribution now covers more of the axis.

Predict before running: If the gap was caused by overfitting to a narrow training distribution, widening it should shrink the held-out drop without changing the training-split score much. The policy sees more variation during training and learns a more general strategy.

If the gap persists, the variation axis is probably not the binding constraint. The next suspects are representation (the policy cannot express the general strategy), exploration (the policy never visits the states needed to learn it), or algorithm choice.

This turns the exercise from a measurement into a test of a hypothesis about why the policy failed to transfer.

What to Do With Your Table

Never report a generalization gap without the seed spread that gives it meaning. A gap of 200 points sounds alarming until you know the seed spread is 180. A gap of 50 points sounds trivial until you know the seed spread is 10.

The decision rule is simple: compare the gap to the noise, not to zero.

If your gap is within noise, your next step is more seeds, not a new algorithm. If your gap is consistent, your next step is the widening experiment. If your gap is seed-dependent, your next step is diagnosing training instability before you touch the environment.

The table you built is the artifact. It tells you what you know, what you do not know, and what to test next. That is the difference between guessing and measuring.