Should This Project Use Deep RL? Practice Choosing With Project Scenarios

You have the checklist memorized. State size. Data cost. Observability. Action structure. Baselines. Debugging burden. Then a real project lands on your desk, the state description is vague, the simulator is slow, and someone wants "AI." You freeze.

That freeze is not a knowledge gap. It is a modeling error. You are treating the six criteria as a scorecard to tally when they are really a set of constraints that can veto each other. The decision is usually made by the first binding constraint—the one that blocks every downstream step until you resolve it. This article gives you four contrasting scenarios, but with a twist: you commit to a verdict before I show you mine. That is the difference between recognizing good reasoning and being able to produce it.

If you need the full decision frame first, the prerequisite guide covers it. Here we practice applying it under messy, conflicting constraints.

Why the Checklist Fails on Real Projects

The criteria interact. A huge state space is survivable if data is cheap. A tiny state space is fatal if you cannot observe the state at all. Tallying them as independent scores hides exactly the interaction that decides your project.

So separate two things:

• A constraint is a hard limit that vetoes deep RL. Expensive exploration is a constraint. Unobservable state is a constraint.
• A cost is something you pay but can afford. Sample inefficiency is a cost when a simulator is cheap.

Two failure modes show up constantly. The first is reaching for deep RL because the problem sounds impressive. The second is refusing it because tabular RL felt safer in a tutorial. Both are vibes, not verdicts.

One clarification before we start, because it trips up almost everyone: the binding constraint is the first blocker you must resolve, not necessarily the one that determines your final method. In Scenario 4 below, observability blocks you before the baseline question even becomes meaningful. Fix the observation, and the baseline gap becomes the next binding constraint. The rule is sequential, not singular.

How to Read a Scenario Before You Choose

Run this procedure every time. It isolates the bottleneck before you touch an algorithm.

1. Write the state as the agent actually receives it. Pixels, a sensor vector, or a handful of integers. Not the state you wish you had.
2. Estimate one episode's cost, then multiply by the order of magnitude of episodes you can afford.
3. Ask whether the state is fully observed. Partial observability changes the method question before it changes the algorithm.
4. Classify the action structure. Discrete and small, discrete and large, or continuous.
5. Name the baseline you must beat. No baseline means no decision, only a demo.
6. Estimate debugging burden. Can you inspect a rollout, replay a failure, and attribute it to a cause?

State the output as one sentence: "The binding constraint is X, therefore Y." That sentence is the whole skill.

For each scenario below, I will give you the setup and then stop. Write your verdict and the constraint that drove it before you scroll to the resolution. The commit is the practice.

Scenario 1: A Small Discrete Control Problem

Setup. A handful of states, a few discrete actions, a cheap simulator, and a fully observed state. Think a grid-world routing task or a small inventory policy.

Your turn. Name the binding constraint and your verdict. One sentence.

Resolution. Start with tabular value methods or even a hand-written policy. Deep RL adds function-approximation failure modes—correlated data, moving targets—without buying generalization you actually need.

Justification. The deciding criterion is state size. Data cost and observability were comfortable, not decisive. Say that out loud, because it trains you to separate the reason from the comfort. The debugging burden here is also trivial: you can print the entire value table and read it. That is a real advantage, not a beginner's crutch.

Common mistake: Assuming deep RL is "more advanced" and therefore more likely to succeed. More machinery means more ways to fail silently.

Scenario 2: High-Dimensional Observations With a Cheap Simulator

Setup. Camera or high-dimensional sensor input, continuous or large discrete actions, and a simulator that generates experience fast.

Your turn. Same drill. Constraint, verdict, one sentence.

Resolution. Deep RL is defensible here. This is the clearest case of when to use deep reinforcement learning: the network's job is representation, not magic. It compresses pixels or sensor streams into features a policy can act on.

Justification. Two criteria cooperate. State size forces function approximation, and cheap data pays for it. Note the residual risk: correlated data and moving targets still apply, so stabilization choices matter even after the decision is made. The debugging burden is higher than Scenario 1—you cannot read a neural network's weights like a table—but the cheap simulator lets you replay failures and inspect rollouts, which keeps the burden manageable.

Scenario 3: Expensive or Risky Real-World Data

Setup. A physical process, a clinical or financial setting, or any environment where each episode is costly, slow, or carries real downside.

Your turn. Constraint, verdict, one sentence.

Resolution. Prefer offline data, supervised or imitation approaches, planning with a model, or a much smaller policy class before committing to deep RL.

Justification. The state space might look impressive enough to tempt you, but it does not matter. When each episode is expensive, sample inefficiency stops being a tuning problem and becomes a budget problem. Action structure is a secondary consideration here—even if the actions are continuous and would normally favor policy-gradient methods, the data cost vetoes the whole family before action structure gets a vote.

Warning: "We will just simulate it" is a claim that must be tested, not assumed. Simulator fidelity is its own project, and a bad simulator quietly relocates your constraint instead of removing it.

Scenario 4: Partial Observability and a Weak Baseline

Setup. The agent sees a noisy or incomplete observation, the action space is moderate, and no credible baseline exists yet.

Your turn. Constraint, verdict, one sentence. Then answer: what is the second binding constraint after you resolve the first?

Resolution. Your first move is not an algorithm choice. It is fixing the observation or building a baseline. You cannot evaluate a deep RL policy against nothing.

Justification. Observability is the first binding constraint: the state you need is not the state you get. But the baseline gap is a second, independent blocker. Even after you fix the observation, a weak baseline turns every result into an unfalsifiable claim. "It seems better" is not a result. There is also an honest middle here: a smaller method with a memory mechanism may beat deep RL, and you should be willing to say so.

Comparing the Four Verdicts Side by Side

Scenario	First binding constraint	Verdict	Second constraint (if any)
Small discrete control	State size (tiny)	Tabular or hand-written policy	None
High-dimensional, cheap sim	None blocking	Deep RL is defensible	Stabilization (correlated data, moving targets)
Expensive real-world data	Data cost	Offline, imitation, or planning	Action structure (secondary)
Partial observability, weak baseline	Observability	Fix observation first	Baseline gap

Notice that data cost appears twice: as a cost in Scenario 2 and a veto in Scenario 3. Same criterion, different role, opposite verdict. No scenario was decided by "deep RL is better" or "deep RL is worse"—only by constraints.

Use this table as a template. Fill in the same columns for your own project.

Your Turn: Run the Procedure on Your Own Project

Write your project's state as the agent receives it. Then estimate episode cost and the number of episodes you can afford. Name the first binding constraint in one sentence and state the verdict it implies. Then name the second constraint, if any.

Here is your self-check: if your verdict changed when you imagined a cheaper simulator, your binding constraint was data cost, not state size. That single test catches most misdiagnoses.

Then run the smallest version of your chosen method and measure whether it beats the baseline you named. If you cannot name a baseline, you are not ready to choose a method—you are ready to build one.

The decision rule to carry: find the first binding constraint, name the baseline second, and only then choose a method. The skill you are practicing is justification under constraint, not algorithm selection. Pick one real project this week, run the procedure, and write the one-sentence verdict. Let your constraint—not the most fashionable algorithm—point you toward the method family you learn next.