Test When Curiosity Rewards Distract an RL Agent

Watch the two curves long enough and you will see the moment curiosity stops working for the agent and starts working for itself. The intrinsic bonus keeps climbing. Novelty refuses to decay. Task reward flatlines. Nothing crashed, no exception was thrown, and the agent is still learning — just not the thing you asked it to learn.

This is a controlled experiment for that moment. You will build a tiny task with a controllable distractor, run it with and without a prediction-error bonus, and log external reward next to distractor visits so the two failure signatures become visible in numbers instead of vibes.

What You Are Actually Testing

The claim under test is narrow: a prediction-error bonus can divert behavior from task progress when a controllable novel distractor exists. Not "curiosity is bad." Not "intrinsic rewards don't work." Just this — under these conditions, the bonus can quietly become the objective.

Two failure signatures matter:

• Task reward stalls or falls while distractor visits rise.
• The intrinsic bonus stays high while external reward never recovers.

Both signatures share a cause. A prediction-error bonus is a signal, not a goal. It exists to exhaust novelty so the agent can get on with the task. When novelty never exhausts — because the agent's own actions keep generating new surprises — the signal becomes a destination. Same code, different outcome. The experiment is how you tell which one you built.

The Toy Task and Its Controllable Distractor

The environment is deliberately small: a discrete grid with a task goal in one corner and a separate distractor region the agent can choose to enter. Nothing exotic. The point is not the grid; it is the distractor's design.

A controllable distractor is one whose state changes in response to the agent's own actions. That property is the whole experiment. A static distractor becomes boring fast — the agent predicts it, prediction error collapses, and the bonus self-limits. A controllable distractor keeps generating prediction error because every action inside it produces a new state to be surprised by.

Two knobs you will vary:

• Distractor novelty — how often its state changes when the agent acts inside it.
• Distractor reward — whether entering it pays any external reward.

Those two knobs separate "distracted" from "correctly redirected." A distractor that pays external reward is not a distraction at all; it is part of the task. Keep that distinction in mind when you read the sweep results.

Setup, Dependencies, and What You Need

You need Python 3.9 or newer, NumPy, and Matplotlib. No deep-learning framework. The base learner is tabular Q-learning so the intrinsic bonus is the only variable under study — if you add a neural network, you have two things changing at once and no clean attribution.

Fix a random seed and an episode budget. Every run in this article uses the same seed and the same number of episodes so the curves are comparable.

The critical implementation detail: the prediction-error bonus is added to the reward before the Q-update, not to the environment's reward. The environment stays honest. The agent's learning target gets the bonus. That separation is what lets you log external reward and intrinsic bonus as two distinct quantities.

import numpy as np

def q_update(Q, s, a, r_total, s_next, alpha=0.1, gamma=0.95):
    best_next = np.max(Q[s_next])
    Q[s, a] += alpha * (r_total + gamma * best_next - Q[s, a])
    return Q

r_total is where the bonus enters. Everything downstream depends on what you put there.

Run 1: Task Reward Only

Train with the intrinsic bonus switched off. Log two things per episode: external reward and distractor visits. That pairing is the core measurement of the entire article — a single reward curve tells you almost nothing about where the agent spent its time.

Expected shape: task reward rises, distractor visits fall as the policy converges. The agent pokes around early, finds the goal, and settles.

def run_episode(env, Q, epsilon, use_bonus, beta=0.0):
    s = env.reset()
    total_ext, distractor_visits = 0.0, 0
    done = False
    while not done:
        a = epsilon_greedy(Q, s, epsilon)
        s_next, r_ext, done = env.step(a)
        if env.in_distractor(s_next):
            distractor_visits += 1
        r_total = r_ext
        if use_bonus:
            r_total += beta * env.prediction_error(s, a, s_next)
        Q = q_update(Q, s, a, r_total, s_next)
        s = s_next
        total_ext += r_ext
    return Q, total_ext, distractor_visits

Why the baseline matters: without it, a later drop in task reward has no reference point. You cannot call something a distraction until you know what undistracted looks like.

Run 2: Add the Prediction-Error Bonus

Now set use_bonus=True. The bonus is prediction error on the distractor's state transition, scaled by a coefficient beta. Log intrinsic bonus, external reward, and distractor visits on the same episode axis.

Two patterns to watch for:

• Healthy: intrinsic bonus decays as novelty is exhausted while external reward climbs. The bonus did its job — it pulled the agent through unexplored territory and then got out of the way.
• Warning: intrinsic bonus stays high, distractor visits stay high, external reward plateaus or drops. The bonus is no longer a means to an end. It is the end.

The healthy pattern is what you want. The warning pattern is what this experiment exists to make visible. If you only plot total reward, the two patterns blur into one ambiguous curve — which is exactly why the logging setup separates them.

Vary the Distractor: Novelty and Reward Settings

One comparison gives you an anecdote. A small sweep gives you a boundary. Run three settings with the same seed and budget:

Setting	Distractor novelty	Distractor reward	Expected outcome
A	High	Zero	Classic distraction: visits rise, task reward stalls
B	Low	Zero	Prediction error collapses, bonus self-limits, task reward recovers
C	High	Positive	Visits rise but external reward rises too — redirected, not distracted

Setting A is the failure case. Setting B shows the bonus behaving correctly because the distractor stops being surprising. Setting C is the one that trips people up: the agent spends time in the distractor region, but its external reward climbs because the distractor pays. That is not distraction. That is the agent correctly identifying where value lives.

The decision rule that falls out of the sweep: distraction is a function of persistent prediction error, not of the bonus existing at all. The bonus is innocent. Persistent novelty is the culprit.

Failure Modes and Debugging Signals

When your own run looks wrong rather than interesting, check these in order.

Bonus scale too large. If beta dominates the Q-update, the intrinsic term swamps the task signal and task reward never moves. Start small and scale up until you see the effect you are testing for.

Bonus never decays. The distractor's state keeps changing, so novelty never exhausts. Check whether the distractor is genuinely controllable — if the agent's actions do not affect it, prediction error should collapse on its own.

Reward normalization missing. Intrinsic and external rewards on different scales make beta meaningless. Normalize both before combining them, or you are tuning a coefficient against an unknown ratio.

Misaligned objectives. If external reward falls while intrinsic rises, the two signals are competing rather than cooperating. That is the signature this whole experiment is designed to surface.

Logging mistakes that hide the problem. Averaging across seeds blurs the failure signature into a smooth line. Plotting only total reward instead of the two components separately does the same thing. Keep the curves separate.

Common mistake: Treating a high intrinsic bonus as evidence the agent is exploring well. A high bonus that never decays is evidence the agent found something it cannot stop being surprised by.

One Modification to Try Next

Add a decay schedule to the intrinsic bonus and rerun Setting A. Multiply beta by a factor that shrinks each episode, or gate the bonus so it cannot exceed a fixed fraction of external reward.

Predict the outcome before you run it. If the diagnosis is right — persistent prediction error is the problem — then decaying the bonus should let external reward recover even though the distractor is still controllable. Distractor visits drop. External reward climbs. The intrinsic bonus fades on schedule instead of on novelty.

If that prediction fails, the diagnosis was incomplete, and you have a better experiment to run next.

Where This Leaves You

A prediction-error bonus is a tool for exhausting novelty, not a permanent objective. The moment it stops decaying is the moment it starts competing with the task. That single rule travels to any intrinsic-reward setup you build: keep the two curves separate, watch for the divergence, and trust the divergence over the bonus value itself.

The next practical step is to apply the same logging pattern to a sparse-reward task where the bonus is supposed to help. If the two curves cooperate there — intrinsic decaying, external climbing — you have confirmed the mechanism works when novelty is genuinely exhaustible. If they diverge, you have found the same boundary in a new environment, and you already know what to check first.