Inverse Reinforcement Learning: Inferring Goals From Demonstrations

Demonstrations tell you what an expert did. Inverse reinforcement learning asks why—and the answer is never uniquely determined by the behavior itself.

The Reversal: From Reward to Behavior, Then Back

Standard reinforcement learning starts with a reward function and asks a forward question: which policy maximizes expected return? You define the objective, then search the space of behaviors for the one that best satisfies it.

Inverse reinforcement learning (IRL) runs the question in reverse. Given expert trajectories—sequences of states and actions—you infer the reward function that would make that behavior optimal. The reward is the hidden objective the expert was optimizing, and your job is to recover it from the observable trace of their choices.

You already know the machinery this assumes. An MDP gives you states, actions, transitions, and a discount factor. IRL keeps all of that intact and declares the reward function unknown. The expert's policy is treated as evidence about that missing piece.

Think of IRL as finding the problem from the solution. Forward RL says: here is the objective, discover the behavior. IRL says: here is the behavior, discover the objective. That framing sounds clean. The trouble begins when you ask whether the objective is actually discoverable.

Why Demonstrations Alone Cannot Reveal the Reward

Here is the uncomfortable structural fact at the heart of inverse reinforcement learning: infinitely many reward functions can rationalize the same observed behavior.

The condition for a reward to explain a demonstration is modest. Any reward function that assigns the demonstrated policy a value at least as high as every alternative policy makes that behavior look optimal. That is a large set. In fact, the degenerate case is always available: a constant reward makes every policy equally good, which means it trivially explains any demonstration you give it.

The ambiguity is not a bug in a particular algorithm. It is a property of the inverse problem itself.

Consider an agent that always takes the shortest path through a maze. What is it optimizing? Speed, certainly—but also energy expenditure, or risk avoidance, or any weighted combination of those objectives that happens to favor the same route. The trajectory is identical in every case. The behavior cannot distinguish between the rewards that produced it.

This is why IRL reward ambiguity is not something you fix with more data. Additional demonstrations can rule out some reward functions, but they cannot collapse the space to a single objective. The underdetermination is structural: behavior is a projection of the reward, and projections lose information.

The Inference Loop: What IRL Actually Does

To see why ambiguity persists, it helps to watch the mechanism IRL methods share. Strip away the mathematical packaging and every classical IRL algorithm runs the same loop:

1. Choose a reward representation. Decide what the reward can look like—typically a set of features you believe matter, or a parameterized function family.
2. Propose reward weights. Start with a guess about how those features should be weighted.
3. Solve the forward problem. Run a standard RL algorithm to find the optimal policy for your proposed reward.
4. Compare behavior. Measure how closely that policy's behavior matches the demonstrations.
5. Update the reward. Adjust the weights to reduce the gap, then repeat.

Notice what the loop compares: behaviors, not rewards. The algorithm never sees the expert's true objective. It only sees whether the policy induced by its current reward hypothesis produces trajectories that look like the demonstrations. If two different rewards both induce policies that match the demonstrations, the loop has no basis to prefer one over the other.

That is why the ambiguity is inescapable. The comparison step is the only source of signal, and behavior is a lossy projection of reward.

IRL vs Imitation Learning: Objectives, Not Actions

If demonstrations cannot uniquely reveal a reward, why bother trying? Why not just copy the demonstrated actions and move on?

Behavior cloning does exactly that—and it inherits a specific failure mode. When the learned policy makes a small error, it drifts into states the expert never visited, and then it has no idea what to do. The errors compound over the trajectory. You have seen this pattern before: supervised imitation fails off the training path.

IRL offers a different kind of generalization—conditionally. Instead of copying actions, it recovers an objective that can then be optimized in states the expert never encountered. If a driving demonstration implies that avoiding collisions matters, the inferred reward may generalize to an unseen obstacle—a deer crossing the road, a construction zone—even though no demonstration showed that exact situation.

That conditional is doing real work. The reward only generalizes if its features and assumptions capture the structure that actually matters. If the inferred reward weights speed heavily and treats collision avoidance as a secondary term, optimizing it in an unseen scenario can produce behavior the expert would never choose. The same demonstrations, under different assumptions, can yield a reward that generalizes beautifully or one that fails dangerously. IRL does not guarantee off-demonstration competence; it makes that competence possible when the reward representation is right.

That is the real argument for IRL: it learns why the expert acts, not just what the expert does. The trade is computational. IRL trades the simplicity of supervised imitation for a much harder inference problem, and most methods pay for that by solving a full RL problem repeatedly as an inner loop.

How IRL Methods Resolve the Ambiguity

Since the ambiguity cannot be removed by data alone, IRL methods resolve it through assumptions. The choice of assumption shapes which reward the method recovers.

Before comparing families, one term helps: state-action occupancy is simply a record of how often the agent visits each state and takes each action. Matching occupancy means matching the overall pattern of behavior, not just individual actions.

Feature-based IRL assumes the reward is a linear combination of known features. The method then searches for feature weights that make the expert's policy clearly better than alternatives. The features encode your prior beliefs about what might matter—safety, speed, efficiency—and the learning problem reduces to weighting them. This is the classic Ng–Russell formulation, and it resolves ambiguity by constraining the reward to a specific functional form.

Maximum-entropy IRL takes a different stance. Instead of committing to a single reward, it models the expert as choosing trajectories with probability proportional to their return. The recovered reward is the one that spreads probability as evenly as possible across behaviors consistent with the demonstrations. This softens the optimality assumption: the expert is not perfectly optimal, just more likely to prefer higher-return trajectories.

Adversarial approaches blur the line between IRL and imitation entirely. Methods like GAIL skip an explicit reward representation and directly match the expert's state-action occupancy. The result behaves like IRL—it can generalize where behavior cloning fails—but it never produces a reward function you can inspect.

Method family	Key assumption	Returns an explicit reward?	Ambiguity resolved by
Feature-based IRL	Reward is a linear combination of known features	Yes	Restricting the reward's functional form
Maximum-entropy IRL	Expert prefers higher-return trajectories probabilistically	Yes	Modeling expert noise and spreading probability
Adversarial approaches	Matching state-action occupancy is sufficient	No	Bypassing reward recovery entirely

The recurring cost across all of these families is the same: most IRL methods solve a full RL problem per iteration. That is why IRL is expensive relative to supervised imitation, and why the computational burden grows so quickly with problem size.

One more point deserves emphasis. The features and priors you choose are not neutral technical details. They are the assumptions that resolve the ambiguity. Different feature choices yield different rewards from the same demonstrations. The method does not discover the expert's true objective; it finds the reward that best fits your assumptions and the observed behavior.

When IRL Helps and When It Misleads

IRL is worth the cost when the reward is genuinely hard to specify by hand. Driving behavior, robotic manipulation, and human preference all fall into this category. You can describe what good behavior looks like, but you cannot write down a reward function that captures it. Demonstrations are the only practical specification of the objective.

Skip IRL when a hand-written reward is adequate, or when you only need to match demonstrated behavior. If your reward is easy to define and your states are well understood, IRL adds cost without adding value.

The decision rule needs more than one condition, though. IRL earns its cost when all of the following hold:

• Off-demonstration decisions matter. You will encounter states the expert never showed you, and behavior cloning's compounding errors would be unacceptable there.
• The environment supports optimization and evaluation. You have a simulator or model you can query, because IRL's inner loop needs to solve forward RL problems repeatedly.
• Your reward representation captures the relevant distinctions. The features you choose must be able to express what actually drives the expert's choices. If they cannot, no amount of inference will recover a useful objective.

When those conditions are absent—no usable environment, no meaningful features, or direct behavior matching is the actual goal—imitation is the honest choice. IRL can add computational cost without producing a trustworthy objective.

The failure modes deserve equal attention.

Reward hacking is the most dangerous. An inferred reward that matches the demonstrations can still reward unintended behavior when optimized further. The reward is an approximation of the expert's objective, and approximations have edges. Push hard enough and you find them.

Suboptimal or noisy experts mislead the recovery process. IRL assumes near-optimal behavior. When demonstrations come from imperfect humans or noisy sensors, the inferred reward can reflect the expert's errors rather than their intent.

Sensitivity to priors means the same data can produce very different rewards under different assumptions. The recovered reward is a hypothesis about the expert's objective, not ground truth. Treat it as an explanation of behavior, not as access to the expert's internal mental state.

The Mental Model That Sticks

Here is the compressed lesson: demonstrations tell you what the expert did, not why. IRL is the disciplined attempt to infer the why—and it always requires assumptions to close the gap between behavior and intent.

The most useful experiment you can run is not a new algorithm. Hold a set of demonstrations fixed, then vary the reward assumptions—different features, different optimality models—and watch how the resulting policies diverge. Some will match the demonstrations closely; others will match them and then fail spectacularly in slightly altered states. That divergence is the underdetermination made visible.

When you can predict which assumptions will produce which failure modes, you understand IRL better than someone who has only memorized its taxonomy. The ambiguity is not a problem to solve. It is the structure of the problem itself—and the assumptions you bring are the only thing standing between behavior and intent.