Experiment With Behavior Cloning and Covariate Shift in Python

A cloned policy can score 100% on the states it trained on and still walk off the path the moment it drives itself. That gap is the whole experiment.

You have probably read the mechanism already: behavior cloning trains a policy on states the expert visited, and the moment the policy makes a small mistake, it lands in a state the expert never showed it. From there, the next mistake is easier, and the one after that is easier still. Covariate shift is the name for it. Compounding error is what it does.

Reading that explanation is not the same as watching it happen. So let us build the smallest sequential task that can expose the drift, train a clone on it, and measure how far the clone wanders from the expert path. Then we will add a correction loop and see whether the drift curve bends.

What We Are Actually Measuring

Before any code, name the claim under test: a policy trained only on expert states degrades once its own actions decide which states it visits.

That claim is easy to assert and easy to hand-wave. To make it observable, we track two numbers across a rollout:

• Per-step action error — how often the clone's chosen action differs from what the expert would have chosen at the same state.
• State distance from the expert path — how far the clone's visited states drift from the states the expert would have visited from the same start.

The first number is the one a supervised-learning mindset reaches for. The second is the one that actually predicts failure. They can disagree sharply, and that disagreement is the lesson.

A tiny deterministic task beats a realistic one here. We are not chasing a benchmark score. We want to see the mechanism move, and a small task lets us print every state and read the whole trajectory in one screen.

Note: This is a controlled toy comparison. It is evidence about the mechanism, not evidence about any real robot, vehicle, or production system. The dynamics here are simple enough that the drift is legible. Real systems add noise, partial observability, and expensive expert queries, and those change the picture.

Setup: A Tiny Sequential Task and an Expert

We need three things: an environment where each action shapes the next state, an expert rule we can read line by line, and a way to collect demonstrations.

Here is the smallest version I would reach for. The agent lives on a one-dimensional line. Its state is its position. Its action is a step of -1, 0, or +1. The goal is to reach position 10 and stay there. The catch: the environment pushes the agent back toward zero with a force that grows with distance, so the agent has to keep pushing forward just to hold its ground.

import numpy as np

np.random.seed(0)

GOAL = 10.0
HORIZON = 40

def step(state, action):
    # action is -1, 0, or +1
    drift = -0.1 * state          # environment pulls back toward zero
    return state + action + drift

def expert_action(state):
    # The expert pushes forward until it reaches the goal, then holds.
    if state < GOAL:
        return 1
    return 0

Read the expert rule once and you know everything it does: push forward until you reach the goal, then hold. There is no hidden policy, no black box, no "correct action" we have to trust. That matters, because the whole experiment is about comparing the clone's action to the expert's, and we need that comparison to be honest.

Now collect demonstrations by rolling the expert out and recording every (state, action) pair it produces.

def rollout(policy, start=0.0):
    states, actions = [], []
    s = start
    for _ in range(HORIZON):
        a = policy(s)
        states.append(s)
        actions.append(a)
        s = step(s, a)
    return np.array(states), np.array(actions)

expert_states, expert_actions = rollout(expert_action)
demo_X = expert_states.reshape(-1, 1)
demo_y = expert_actions

Dependencies are plain Python plus NumPy. No deep-learning framework, no GPU, no external service. The fixed seed keeps the run reproducible, and there is no randomness in the dynamics, so two runs of the same code produce the same numbers.

Train the Clone and Check It on Expert States

The clone is a classifier: given a state, predict the expert's action. A shallow decision tree or a small logistic regression is plenty. I will use a simple nearest-neighbor rule so the fit is transparent.

def clone_policy(state):
    # Predict the expert action of the nearest demonstrated state.
    idx = np.argmin(np.abs(demo_X[:, 0] - state))
    return demo_y[idx]

train_preds = np.array([clone_policy(s) for s in demo_X[:, 0]])
train_acc = np.mean(train_preds == demo_y)
print(f"Training accuracy on expert states: {train_acc:.2f}")

Expected output:

Training accuracy on expert states: 1.00

Perfect accuracy. And that number is almost worthless.

Here is the trap. The demonstration states are the ones the expert already handles well. The clone is being graded on the easiest possible distribution — the states where the correct action is already obvious from the nearby examples. High accuracy there tells you the clone memorized the expert's path. It tells you nothing about what happens when the clone's own actions carry it somewhere the expert never went.

Common mistake: Treating held-out accuracy on demonstration states as a proxy for policy quality. It is a proxy for fit on the training distribution. The policy will not stay inside that distribution once it acts on its own.

Roll Out the Clone and Watch the Path Drift

Now let the clone drive. Same start, same horizon, but the actions come from the clone instead of the expert.

clone_states, clone_actions = rollout(clone_policy)
expert_states, _ = rollout(expert_action)

for t in range(HORIZON):
    dist = abs(clone_states[t] - expert_states[t])
    print(f"step {t:2d}  clone={clone_states[t]:6.2f}  "
          f"expert={expert_states[t]:6.2f}  dist={dist:5.2f}")

The first few steps look fine. Then the distance starts to grow, and it does not come back down. The clone is not failing because it is badly fit — it is failing because it is now being asked about states it never trained on.

Two failure signals show up in that printout, and they are worth separating:

• The first wrong action. At some step, the clone picks an action the expert would not have. That is the seed of the drift.
• The compounding that follows. Every subsequent state is a little further from the demonstrated path, so the nearest-neighbor lookup gets less reliable, so the next action is more likely to be wrong.

The first signal is a single step. The second is a trajectory. Behavior cloning compounding errors is the second one, and it is the one that decides whether the task succeeds.

If you plot state distance against step number, you get a curve that starts near zero and bends upward. That curve is your diagnostic. A flat curve means the task is forgiving. A curve that climbs and never recovers means the policy has left the demonstrated path for good.

Add Expert Correction and Compare

The fix is to stop training only on states the expert prefers and start training on states the learner actually creates. This is the idea behind DAgger-style expert queries: roll out the current policy, ask the expert for the correct action at each visited state, add those pairs to the dataset, and retrain.

In plain terms, the loop is:

1. Roll out the current policy.
2. At every state it visits, ask the expert what the correct action is.
3. Add those (state, expert_action) pairs to the training set.
4. Retrain the clone on the enlarged dataset.
5. Repeat.

def dagger_round(policy, X, y, start=0.0):
    states, _ = rollout(policy, start=start)
    new_X = states.reshape(-1, 1)
    new_y = np.array([expert_action(s) for s in states])
    return np.vstack([X, new_X]), np.concatenate([y, new_y])

X, y = demo_X, demo_y
for round_idx in range(3):
    X, y = dagger_round(clone_policy, X, y)
    # rebuild the clone on the enlarged dataset
    demo_X, demo_y = X, y
    print(f"round {round_idx + 1}: dataset size = {len(y)}")

After a few rounds, re-run the same rollout measurement and compare the drift curve. The expected shape of the result: divergence is slower, the clone stays closer to the expert path for more steps, and task performance improves. The improvement depends on how many correction rounds you run and how much of the state space the learner actually reaches.

Why does this attack the root cause? Because the training distribution is now pulled toward the states the learner creates, not the states the expert prefers. The clone is being taught about the shoulder of the road, not just the pavement.

Warning: Expert queries are cheap in this toy task and expensive in reality. Every correction round in a real system means a human or a simulator labeling the states your policy just invented. That cost is the practical price of the fix, and it is why imitation alone is often paired with environment feedback rather than trusted on its own.

Failure Modes and Debugging Signals

When your own run does not behave as expected, check these first.

The honest limit sits in the second row. Expert correction works because it targets the states the learner creates. If your learner never creates new states — because the task is too easy or the horizon too short — there is nothing for the correction to fix, and the loop looks useless. That is not a failure of the method. It is a failure of the experiment to stress the mechanism.

One Change to Try Next

Turn this into a reusable probe by varying the factor that controls how fast errors compound: the sensitivity of the dynamics.

Change the drift coefficient from -0.1 to -0.5 and predict what happens to the drift curve before you run it. My guess: the clone leaves the expert path faster, the correction loop needs more rounds to help, and the gap between plain cloning and corrected cloning widens. Run it and check.

Then compare a forgiving task against a brittle one. When the dynamics are gentle, covariate shift is a nuisance you can mostly ignore. When they are sharp, it becomes a failure. That boundary — where drift stops being a curiosity and starts being the reason your policy fails — is the thing worth knowing.

The decision rule to carry forward: if your policy's accuracy is measured only on demonstration states, you have not tested it. Measure the drift curve instead. It tells you whether your clone can survive its own first mistake, and that is the question that actually matters.