Build and Verify a Tiny Tabular RL Environment in Python

A learning algorithm that never improves is usually blamed for the wrong crime. More often, the environment quietly hands back the wrong next state, pays the reward one step late, or forgets to end the episode at all. The agent then learns exactly what the environment teaches it — including the lies.

So before we attach any learner, we build a small world and try to break it. This is the first thing I verify in any RL project, and it is the cheapest hour you will ever spend.

Why Verify the Environment Before the Algorithm

The environment is the ground truth your agent learns against. If it lies, the agent learns the lie, and no amount of tuning will fix it. A rising loss curve on a broken environment is just a confident student memorizing a wrong textbook.

The good news: a tabular environment has a small, countable set of states, and a deterministic one always produces the same next state for the same action. That means you can trace every transition by hand. Verification is not a research problem here — it is a checklist.

If you have already worked through how to design an RL environment conceptually — states, actions, rewards, termination — this article makes that contract concrete. We assume you understand the shape of the contract. Now we write it, run it, and prove it holds.

What We Are Building: A 1D Corridor Task

Picture a line of cells numbered 0 to N-1. The agent starts at the far left and must reach the far right. One integer describes the whole world.

• State: a single integer, the current cell.
• Actions: 0 = move left, 1 = move right.
• Reward: a small negative step cost on every move, a positive reward on reaching the goal.
• Termination: the episode ends the moment the goal is reached.

Why this task? It is small enough to trace by hand, yet it still contains the two things beginners get wrong most often: delayed reward and a terminal state. If you can verify this corridor, you can verify a gridworld.

Set Up the Project and Dependencies

You need Python 3.10 or newer and nothing else. No NumPy, no deep-learning framework. One file is enough, and that is deliberate: a framework hides the mechanism we are trying to inspect. We want the transition logic visible, not buried.

We will implement the same interface that common RL libraries use, so the habit transfers later:

• reset() returns the initial state.
• step(action) returns (next_state, reward, done).

Note: Real libraries often return a fourth info dictionary. We skip it here to keep the return tuple readable, but the reset/step shape is the same idea.

Write the Environment Class

Here is the whole thing. Read it once, then we will walk through the decisions.

class Corridor:
    def __init__(self, size=5, step_cost=-0.01, goal_reward=1.0):
        self.size = size
        self.start = 0
        self.goal = size - 1
        self.step_cost = step_cost
        self.goal_reward = goal_reward
        self.state = self.start

    def reset(self):
        self.state = self.start
        return self.state

    def step(self, action):
        # action 0 = left, action 1 = right
        move = -1 if action == 0 else 1
        self.state = max(0, min(self.size - 1, self.state + move))

        if self.state == self.goal:
            return self.state, self.goal_reward, True
        return self.state, self.step_cost, False

A few design choices deserve a sentence each.

reset() must be idempotent. Calling it twice in a row should leave you in the same place. If it does not, episodes are not independent, and your agent carries stale state from one run into the next.

Boundaries clamp, they do not wrap. When the agent tries to move left from cell 0, it stays at 0. I chose clamping on purpose. Wrapping would teleport the agent from one end to the other, which is a different task entirely. The max/min pair is the whole boundary rule, and it lives in one line so you can audit it.

Reward logic lives in one place. Every branch that returns a reward is visible in step. When reward bugs appear later, you have exactly one function to read.

Smoke Test: Does It Run at All

Before we check correctness, we check that the code executes and returns the shapes we expect.

env = Corridor(size=5)
state = env.reset()
print("reset:", state)

next_state, reward, done = env.step(1)
print("step:", next_state, reward, done)

Expected output:

reset: 0
step: 1 -0.01 False

This proves the code runs. It does not prove the code is correct. That distinction matters — a smoke test catches typos, not logic errors. A wrong reward value still prints happily.

Verify Transitions, Rewards, and Termination

Now we move from "it runs" to "it is right." Write these as assertions so the checks are repeatable, not eyeballed once and forgotten.

# Transition check: each action produces the exact expected next state.
env.reset()
assert env.step(1)[0] == 1
assert env.step(1)[0] == 2
assert env.step(0)[0] == 1

# Boundary check: the agent cannot walk off either end.
env.reset()
assert env.step(0)[0] == 0          # left from the leftmost cell stays put
env.state = env.goal
assert env.step(1)[0] == env.goal   # right from the goal stays put

# Reward check: step cost on ordinary moves, goal reward only on arrival.
env.reset()
_, reward, done = env.step(1)
assert reward == env.step_cost and done is False
env.state = env.goal - 1
_, reward, done = env.step(1)
assert reward == env.goal_reward and done is True

# Termination check: done flips exactly when the goal is reached.
env.reset()
assert env.step(1)[2] is False
env.state = env.goal - 1
assert env.step(1)[2] is True

Each block targets one part of the contract. The transition check confirms the world moves as described. The reward check confirms the step cost applies on ordinary moves and the goal reward fires only on the step that arrives. The termination check confirms done is False mid-episode and True at the goal. The boundary check confirms the agent cannot leave the corridor.

If any assertion fails, you have found a bug before it poisoned a learning run. That is the entire point.

Common Failure Modes and How to Spot Them

These are the mistakes I see most often, and each one has a symptom you can recognize.

Failure mode	Observable symptom
Off-by-one in the goal check	The agent reaches the goal but done never flips; episodes run forever.
Reward on the wrong step	Reward fires one step late; credit assignment shifts and learning crawls.
State leaking across resets	reset() does not clear state; episodes are not independent.
Boundary bug	The agent wraps around or indexes out of range instead of stopping.

The pattern is the same in all four: the environment keeps running, the code keeps printing, and the bug hides until learning mysteriously stalls. The assertions above are your early-warning system.

One Experiment: Change the Reward and Predict the Effect

Now the habit that actually builds skill. Change one thing, predict the outcome, then verify.

Set step_cost to 0.0 instead of -0.01. Before running anything, ask: what changes? With no cost per step, wandering is free. A policy that meanders toward the goal is just as good as one that sprints, because only the goal reward matters. Now set step_cost to -1.0. Suddenly every wasted step is expensive, and the best policy is the shortest path.

env = Corridor(size=5, step_cost=-1.0)
env.reset()
_, reward, done = env.step(1)
assert reward == -1.0 and done is False

Re-run your verification checks after the change. The environment must still honor its contract — the reward value changed, but the transition, termination, and boundary rules did not. This is the loop that matters: change one thing, predict, verify, observe.

What You Now Have and Where to Go Next

You have a verified environment you can trust as ground truth. Every transition, reward, and termination signal has been checked against an explicit expectation, and the checks live in the file as a regression test for every future change.

The next step is to attach a simple value-based learner and watch it improve. Here is the payoff: when the learning curve misbehaves now, the environment is no longer a suspect. The bug is in the learner, and you can debug it directly. Keep those assertions in the file. They are the cheapest insurance you will ever buy against a silent, lying world.