Constrained Reinforcement Learning: Optimizing Reward Without Breaking the Rules

A penalty is a price. A constraint is a wall. Most RL safety problems fail because practitioners build walls out of prices and then act surprised when the agent pays them.

Why a Big Penalty Is Not a Hard Rule

The instinct is understandable. You have a rule—don't crash, don't exceed the power budget, don't enter the restricted zone—so you add a large negative reward for violating it. The agent learns to avoid the behavior. Problem solved.

Except it isn't solved. What you've actually done is change the optimization problem from "maximize reward subject to safety" into "maximize reward minus a safety-weighted cost." Those are different objectives, and the difference shows up exactly where you can't afford it.

Here's the mechanism: a penalty optimizes an expected tradeoff. The agent isn't deciding whether to violate the rule; it's deciding whether the violation is worth it. If the reward gain from one risky action exceeds the expected penalty cost, the agent takes the action. Not because it's broken, but because you told it—through the reward function—that this is acceptable behavior.

The penalty coefficient becomes a hidden dial that trades reward against safety. Crank it too low and the agent violates constantly. Crank it too high and the agent becomes paralyzed, refusing useful actions that carry even a small risk of triggering the penalty. And the right setting depends on the task, the dynamics, and the reward structure in ways you can't know in advance.

Think about the difference between a speed limit sign and a fine. A fine changes the expected cost of speeding; if the fine is low enough or the time savings high enough, speeding remains rational. A hard rule forbids the action entirely. The constrained reinforcement learning literature exists because many real problems need the second kind of guarantee, not the first.

This builds directly on the reward-shaping versus sparse-rewards distinction: shaping changes what the agent is incentivized to do, but it never changes what the agent is allowed to do.

The Constrained MDP: Reward and Cost as Separate Signals

The formal fix is the constrained Markov decision process (CMDP). You keep the standard MDP tuple—states, actions, transitions, rewards, discount factor—and add one or more cost signals alongside the reward.

The objective becomes:

Maximize expected discounted reward, subject to expected discounted cost staying within a chosen budget.

That single change reframes the entire problem. The agent is now judged on two axes. Reward says "how well did you do the task." Cost says "did you respect the rules while doing it." Feasibility becomes a first-class requirement rather than an emergent side effect of getting the penalty coefficient right.

The cost signal is not "negative reward." It is a different measurement that the policy is held accountable to. You can picture reward and cost as two parallel streams flowing from the same state-action pairs into separate accumulators. The agent maximizes one accumulator while keeping the other below a threshold.

The threshold is a design choice with real consequences. Set it too loose and the constraint is meaningless. Set it too tight and no reasonable policy can satisfy it. The threshold encodes your actual safety or resource limit—it should come from the problem, not from a hyperparameter sweep.

Feasibility vs. Discouragement: What a Constraint Actually Guarantees

Here is where the terminology stops being academic and starts determining what you can claim about your trained policy.

A policy is feasible if it keeps cumulative cost within the budget across the states it actually reaches. That sounds straightforward, but the expected-cost formulation hides a subtlety: the constraint is an average, not a per-trajectory guarantee. A policy can be feasible on average while individual episodes break the rule. If you need every episode to respect the constraint—every robot run, every trading day, every delivery—an expected-cost constraint is the wrong tool.

This connects to the idea of a feasible set. Not every state is reachable while respecting the constraint. The agent can only be safe within the region where a compliant policy exists. If the starting state lies outside that region, or if the task demands reaching states that no constraint-respecting policy can reach, the problem is infeasible. No algorithm will rescue you.

The practical consequence is sharp: choosing the wrong formulation changes what "safe" means. A penalty-weighted reward gives you a policy that violates rarely enough to look acceptable in training. A constraint gives you a policy that respects a budget—but only the budget you defined, averaged the way you defined it. Neither gives you a guarantee that every individual episode stays safe. If you need persistent safety—staying inside a safe region at every step—you need a different formulation entirely, one that reasons about reachability rather than cumulative cost.

How Constrained RL Algorithms Actually Enforce the Budget

Once you've formulated the problem as a CMDP, the question becomes mechanical: how do you actually find a policy that maximizes reward while respecting the cost budget? The main algorithmic families answer one question differently: where does the enforcement live?

Lagrangian / primal-dual methods turn the constrained problem into an unconstrained one by adding a learned multiplier that rises when the constraint is violated. The agent maximizes reward minus the multiplier times cost; the multiplier rises until the constraint binds. This is elegant and general, but it reintroduces a version of the penalty-coefficient problem—now the coefficient is learned rather than hand-tuned, and the optimization can oscillate or converge slowly.

Trust-region and policy-search methods, such as constrained policy optimization, take a different approach. Instead of penalizing violations in the objective, they constrain the update step itself: improve reward while keeping the policy within a region where the cost constraint remains satisfied. The enforcement lives in the update, not in the reward.

Safety-layer and projection approaches filter the action space. The agent proposes an action; a safety layer checks it against the constraint and projects it onto the nearest safe action if needed. Enforcement lives in the action space itself—the agent literally cannot propose an unsafe action.

Reachability and feasible-set methods reason about which states can be kept safe over time, rather than only bounding cumulative cost. These are the methods that can provide persistent safety guarantees, but they typically require more knowledge about the dynamics and are harder to apply in complex environments.

These families are not interchangeable. Lagrangian methods are general but give average-case guarantees. Safety layers give hard per-action guarantees but require a reliable safety check. Reachability methods give the strongest guarantees but demand the most from your model of the environment.

When Constraints Are the Right Tool and When They Are Overkill

Here is the decision rule I use:

Use a constrained formulation when a violation is genuinely unacceptable, or when the cost of tuning a penalty coefficient exceeds the cost of enforcing a budget. Stay with a shaped reward when the "constraint" is really a soft preference and occasional violations are tolerable.

The honest test is this: would you accept a policy that violates the rule once in a while for a large reward gain? If yes, you have a preference, not a constraint. If no—if even one violation is a failure—you need a constrained formulation, and you need to think carefully about what kind of guarantee you actually require.

Constraints carry hidden costs. They make optimization harder, add hyperparameters (thresholds, multipliers, trust-region sizes), and can slow or destabilize training. A penalty-weighted reward is often easier to optimize; it just optimizes the wrong thing. The question is whether the wrong thing is close enough to what you need.

This same distinction recurs at different levels. In the invalid-actions context, masking is a hard constraint on the action space while a penalty is a soft one. The principle is identical: if an action must never happen, don't price it—forbid it.

Common Mistakes When Setting Up a Constrained Problem

When you move from penalties to constraints, the failure modes shift. Watch for these:

Treating the cost threshold as a free knob. The threshold should be derived from your actual safety or resource limit. If you tune it like a hyperparameter, you're back to penalty-coefficient thinking with extra steps.

Measuring cost only at the end of an episode. If the constraint is cumulative, you need to track cost along the way. An agent that discovers mid-episode that it has already exhausted its budget has learned something important—but only if the cost signal is visible at every step.

Forgetting that expected-cost budgets permit bad episodes. Match the formulation to the guarantee you actually need. If you need per-trajectory safety, an expected-cost constraint won't give it to you.

Ignoring feasibility. If no policy can satisfy the constraint from the starting state, the problem is infeasible. Check this before training, not after.

Confusing the cost signal with the reward signal. They flow through the same machinery but serve different purposes. Optimizing the wrong quantity is surprisingly easy when both signals are scalar and both are discounted.

The path forward is concrete. Name your reward: what are you maximizing? Name your cost: what are you measuring? Decide whether the requirement is a hard budget or a soft preference. Only then choose between a shaped reward and a constrained formulation.

Write down the CMDP for your own task before you touch an algorithm. The formulation is where the real design happens. The algorithm is just the machinery that solves the problem you actually defined.