Reward Models From Human Preferences: Turning Comparisons Into a Learning Signal

A reward model is a compression. Thousands of noisy human comparisons get squeezed into one scalar function, and the compression itself is where meaning gets lost. The question is not whether the model captures what people meant—it is how much of it survives the squeeze.

What a Reward Model Actually Is

A reward model from human preferences is a parameterized function that maps a context and a candidate outcome to a scalar score. Give it a prompt and a response, and it returns a number. That number is supposed to represent how much a human would prefer that response over alternatives.

Notice what this object is not. It is not a specification of what is good. It is not a rulebook, a rubric, or an explicit statement of values. It is a learned proxy for human judgment, trained to reproduce the patterns present in preference data rather than to encode any underlying principle.

The scalar output is what makes the whole pipeline work. Reinforcement learning algorithms need a reward signal—a number they can maximize. A reward model supplies that number on demand, for any candidate outcome the policy produces. This is why the architecture matters: the model must be able to score novel outcomes, not just the ones humans already judged.

Keep the three objects separate. The policy generates candidate outcomes. The preference data records which outcomes humans preferred. The reward model sits between them, translating the static data into a dynamic scoring function the policy can optimize against. Confusing these roles is where a lot of downstream debugging time goes.

From Comparisons to a Scalar: The Bradley-Terry Step

Humans give relative judgments. Shown two responses, A and B, a person can usually say which one they prefer. But reinforcement learning needs a scalar: a score for A, a score for B, and ideally a score for every other candidate the policy might generate.

The Bradley-Terry model is the bridge between those two formats. It assumes that the probability of A being preferred over B follows a logistic function of the difference in their underlying reward scores:

[ P(A > B) = \sigma(r(A) - r(B)) ]

where (\sigma) is the sigmoid function. If A's score is much higher than B's, the model predicts A wins almost always. If the scores are close, the prediction approaches a coin flip.

Training maximizes the likelihood of the observed preferences. For each pair where a human said "A is better than B," the model pushes A's score above B's. The loss function cares only about the margin between chosen and rejected scores, not their absolute values. A reward model that gives everything a score of 5 is indistinguishable, under this loss, from one that gives everything a score of 50—only the differences matter.

This single assumption carries a heavy load. The Bradley-Terry model assumes that one shared reward function explains all annotators' preferences. It assumes that preferences are transitive: if someone prefers A to B and B to C, they should prefer A to C. It assumes that the probability of preferring A over B depends only on their individual scores, not on the set of alternatives being compared.

Each of these assumptions can fail in practice. When they do, the failure is silent—the model still trains, still produces scores, still looks plausible.

What the Training Data Carries (and Hides)

A reward model can only reflect the variation present in its training data. Preferences nobody expressed are invisible to it. This sounds obvious, but the consequences are easy to underestimate.

Annotator disagreement is not noise. When two humans look at the same pair of responses and disagree about which is better, the standard training procedure treats that as a coin flip and averages over it. But disagreement often signals genuinely divergent preferences—different people value different things. Averaging produces a reward model that matches no single person's actual preference. It invents a composite judge who never existed.

Selection bias shapes what the model can learn. Which pairs get collected determines what comparisons the model can possibly absorb. If the data collection process only samples responses from a narrow distribution, the reward model never sees the full range of outcomes the policy might later produce. The blind spots in the data become blind spots in the reward.

Outcome supervision and process supervision learn different things. An outcome reward model scores final results; a process reward model scores intermediate steps. They answer different questions and fail in different ways. An outcome model can reward a correct answer reached through flawed reasoning. A process model can reward plausible steps that lead nowhere.

The data pipeline is not a neutral conduit for human judgment. It is a filter that selects which judgments get expressed, which disagreements get smoothed over, and which outcomes ever get compared.

Why a Learned Reward Can Be Wrong

The failure modes of reward models are not abstract risks. They are observable patterns with identifiable mechanisms.

Reward hacking is what happens when the policy exploits features the reward model learned to associate with preference, without actually producing what humans wanted. The reward model is a proxy objective, and optimizing a proxy is not the same as optimizing the intent behind it. The policy finds the shortest path to a high score, even when that path diverges from genuine quality.

Verbosity bias is a specific, well-documented instance. Annotators systematically prefer longer responses, perhaps because length correlates with effort or thoroughness. The reward model learns this correlation and starts rewarding verbosity directly. The policy, optimizing the reward model, produces increasingly long-winded output. The reward model is not wrong about its training data—it is wrong about what the data meant.

Surface-feature overfitting is the general case. The model latches onto features that correlate with preference in the training set but do not cause it: formatting artifacts, hedging language, specific vocabulary, response structure. These features are cheap for the policy to produce, so it produces them in abundance.

Annotator averaging produces a reward that represents nobody. When preferences are genuinely multimodal—some users want concise answers, others want detailed explanations—a single scalar reward is structurally unable to represent everyone. The model finds a compromise that satisfies no one fully.

The unifying theme: a reward model learns correlations between outcomes and preferences as expressed in a specific dataset. Correlations are not causes. When the policy optimizes the correlation instead of the underlying intent, the model's blind spots become the policy's exploits.

Reading a Reward Model's Blind Spots

Treating a reward model as a black box is a choice, and it is usually the wrong one. There are concrete ways to probe where a learned reward diverges from what people meant.

Check for cheap correlates. Score responses that differ only in length, formatting, or hedging. If the reward model consistently prefers the longer or more polished version regardless of substance, you have found a bias, not a judgment.

Probe with adversarial inputs. Generate edge cases—unusual but valid responses, deliberately concise answers, responses that game the surface features you suspect. See what the model actually scores highly. The gaps between what it rewards and what a human would reward are your failure map.

Compare against held-out human judgments. Take preference pairs the model never saw during training and check whether its rankings match human rankings. Leaderboard accuracy is a bounded claim about benchmark conditions, not proof of alignment. A model that scores 94% on a benchmark can still fail catastrophically on the specific distribution your policy explores.

Watch for the averaging artifact. If annotator disagreement was high in your training data, ask whether the learned reward represents a real preference or a synthetic compromise. When preferences genuinely differ across people, no single scalar can represent everyone—and pretending otherwise just hides the divergence.

The diagnostic frame is simple: the reward model is a proxy, and proxies need auditing. The question is not whether the model is perfect. It is whether you know where it is wrong before the policy finds out for you.

When a Learned Reward Is the Right Tool

Learned reward models earn their place under specific conditions. The target judgment must be hard to specify but easy for humans to recognize. You cannot write a rule for what makes a good summary, a helpful assistant response, or a natural-sounding dialogue—but you can reliably pick the better of two options. That gap between specification difficulty and recognition ease is where preference learning lives.

When a task has verifiable correctness, a learned reward model is usually the wrong tool. If you can check whether a mathematical proof is valid, whether code compiles and passes tests, or whether a factual claim matches a trusted source, use that verifiable signal instead. A verifier does not hallucinate preferences, does not reward verbosity, and does not average over divergent annotators. It is not a proxy—it is the thing itself.

The cost calculation matters too. Preference data is expensive to collect, annotator disagreement adds noise, and reward hacking is a persistent risk. Every one of those costs needs to be justified by the value of the learned signal. If a simpler approach—an explicit rule, a verifier, direct supervision—can do the job, it will almost certainly do it more reliably.

A learned reward is a proxy. The question is whether the proxy is good enough for the downstream use, not whether it is perfect. Sometimes it is. Sometimes it is not. The mistake is assuming you can tell the difference without looking.

The Decision Rule

Treat a reward model as what it is: a compressible, fallible proxy for human judgment. Audit it against held-out preferences and adversarial probes before you trust it. Choose a verifiable signal whenever the task allows one.

And when you do use a learned reward, do not ask whether it captures what people meant. Ask where it stops capturing it—then go find those places before the policy does.