Collecting Preference Data for RL: Better Comparisons, Better Reward Models

More preference labels do not automatically mean a better reward model. The reward model is only as sharp as the comparisons that feed it.

A pair of responses that every annotator instantly agrees on teaches the reward model almost nothing new. The expensive, valuable comparisons sit near the boundary where two behaviors are close but meaningfully different—where annotators hesitate, weigh tradeoffs, and sometimes disagree. If you treat preference data collection as a volume game, you will spend your annotation budget confirming what the model already knows. If you treat it as a measurement problem, every comparison becomes a probe aimed at a specific unknown.

Why the Comparison, Not the Label, Carries the Signal

A preference label only records a judgment between two options. The information content lives in how those options differ on the dimension you actually care about.

Think about what a reward model learns from pairwise comparisons. It observes which response won and which lost, then adjusts its internal scoring to make the winner rank higher. When you show it a pair where one response is clearly excellent and the other is clearly broken, the model updates toward a conclusion it probably already reached. The label confirms existing knowledge; it does not create new knowledge.

Trivially different pairs are cheap to annotate and nearly worthless to learn from. The comparisons that matter are the ones where two responses are both plausible, both partially correct, but different in a way that reveals the underlying objective. Should the model refuse this borderline request or answer it with a caveat? Should it give the concise summary or the detailed walkthrough? These are the pairs that force the reward model to learn where you actually draw the line.

The collection step determines what the reward model can and cannot represent. No amount of training cleverness can recover information that the comparison set never contained.

What Makes a Comparison Informative

Before spending annotation budget on a pair, run it through a quick triage. Four questions decide whether the comparison earns its cost.

Does the pair differ on the behavior you want to shape? Two responses that differ only in verbosity teach style. Two responses that differ in whether they refuse a borderline request teach the safety boundary. If you want the model to learn when to refuse, you need pairs that force that decision. Incidental surface differences—length, formatting, word choice—consume annotation effort while leaving the objective untouched.

Is the pair valid and in-distribution? A comparison is only useful if both options are plausible outputs from the kind of policy you will eventually run. Strange, degenerate, or off-topic responses create uncertainty the reward model should not spend capacity on. Validity comes before uncertainty: a pair that fails this check should be regenerated, not labeled.

Does the right answer depend on context or tradeoffs? An informative comparison forces the annotator to weigh competing values. Helpfulness against safety. Brevity against completeness. Directness against politeness. When the annotator must make a judgment call, the reward model must learn the priority order behind that call. When the answer is obvious from surface features alone, the model learns surface features instead.

Does the pair add coverage? Pairs should span the full range of behaviors the policy might produce, including edge cases and failure modes. If you only collect comparisons from the comfortable middle—typical requests, well-behaved responses—the reward model becomes confident there and blind at the boundaries. The policy will encounter those boundaries during optimization, and the reward model will have no evidence to guide it.

Here is the concrete version. Two responses to a request for medical advice: one refuses and suggests seeing a doctor, the other provides detailed home remedies. Annotators who prioritize safety prefer the refusal. Annotators who prioritize helpfulness prefer the remedies. That pair forces the reward model to learn where your safety boundary sits—assuming safety-before-helpfulness is the objective you actually want to encode. A pair differing only in how politely the refusal is worded teaches nothing about the boundary itself.

Choosing Pairs: Random Sampling Is a Weak Default

If you generate candidate responses and pair them randomly, most of your comparisons will land in the easy region where annotators agree. The reward model receives redundant evidence about distinctions it already handles and almost no evidence about the boundaries it cannot separate.

Targeted selection fixes this by asking a different question: which pairs would reduce the reward model's uncertainty the most? The answer is usually the pairs it cannot currently separate—the ones where its predicted preference is close to a coin flip.

This is the logic behind active preference learning. The loop looks like this:

1. Train a provisional reward model on the comparisons you have.
2. Generate candidate responses that span the behaviors you care about.
3. Find the pairs where the provisional model is most uncertain about which response should win.
4. Send those pairs to annotators.
5. Retrain and repeat.

Each iteration pushes annotation toward the boundary that needs disambiguation. Instead of confirming what the model knows, you probe what it does not.

Active selection has a failure mode worth respecting. If you chase model uncertainty without guardrails, you can drift toward adversarial pairs—responses that are strange, degenerate, or irrelevant to real use. The model is uncertain about them precisely because they are outliers. That is why the triage order matters: validity and relevance checks come first, uncertainty second. A high-uncertainty pair that fails the validity check should be regenerated, not annotated. A high-uncertainty pair that passes validity but sits far outside your target behavior distribution should be deferred, not collected. Only pairs that pass relevance and validity deserve annotation budget, and among those, uncertainty selects the order.

Coverage adds the final guardrail. Mix uncertainty-driven selection with periodic samples across the full behavior space, including edge cases, to keep the collection grounded.

The pair is either a valid, in-distribution boundary case or a strange outlier; the team has not checked which.

Annotator Disagreement Is Evidence to Inspect

The instinct to discard disagreement is understandable. It feels like sloppy data. But disagreement is not automatically valuable, and it is not automatically noise. It is diagnostic evidence—and the diagnosis determines your response.

Three different causes produce disagreement, and each demands a different action:

Disagreement pattern	Likely cause	What to do
Systematic split across annotator groups	Genuine value conflict in the objective	Decide whether to specify a shared priority, segment the data by group, or escalate the objective question
Scattered, inconsistent disagreement	Unclear instructions or missing context	Revise the annotation task, add examples, clarify the priority order
One annotator consistently out of line	Careless or invalid labeling	Filter or relabel that annotator's contributions

Consider the medical advice example again. The refusal and the remedies response will split annotators along their values. Some people weight safety above helpfulness; others weight helpfulness above safety. Forcing those annotators to reach consensus does not resolve the underlying value difference. It just hides it.

But here is the question the split forces you to answer: is there one shared objective with a boundary to locate, or is the objective itself contested? If your product genuinely requires safety before helpfulness, the systematic split tells you the boundary is hard to locate and worth more annotation. If different user populations legitimately want different tradeoffs, no single reward model can satisfy everyone, and majority voting just encodes the largest group's values. That is an objective-design decision, not a labeling problem.

The practical move is to keep disagreement flags in the dataset rather than majority-voting them away. When you see systematic disagreement, ask whether the objective itself is contested and whether you need to sharpen your statement of it. When you see scattered disagreement, ask whether the annotation instructions are clear enough. Disagreement is a measurement of the objective's ambiguity—but only careful inspection tells you whether the ambiguity lives in the objective, the instructions, or the annotator.

Designing the Annotation Task and Instructions

The format of the annotation task shapes what annotators can express. A binary forced choice between two responses hides nuance. A scale that includes strength of preference communicates more. And critically, annotators need a way to say "both are poor" or "this is a tie." Forcing a preference between two bad responses injects noise into the reward model—it learns a ranking that does not reflect anyone's actual judgment.

Clear instructions matter just as much as format. If you name the priority order explicitly—safety before helpfulness, for example—annotators produce more consistent labels without erasing genuine value differences. The instruction removes spurious disagreement caused by unclear task framing, not the disagreement caused by contested values.

Multiple annotators per comparison catch labeling mistakes, but only when the disagreement is about care rather than about the objective. That distinction is the whole game. Adjudication resolves sloppy labels. It should not resolve value conflicts by fiat.

Logging What You Learn: One Record per Comparison

Disagreement analysis only works if you log the right fields from the start. A useful comparison record carries more than the final label. Store the signals that let you diagnose the next collection decision:

Field	What it captures
Comparison ID	Links the label to the original pair
Behavior dimension	Which objective boundary this pair probes
Candidate source	Whether the pair came from uncertainty selection, coverage sampling, or random generation
Annotator labels and confidence	The raw judgments, including strength of preference
Model preference probability	What the provisional reward model predicted
Coverage bucket	Which region of the behavior space this pair represents

Here is how one record drives a decision. Suppose a pair shows high model uncertainty, strong annotator agreement, and a coverage bucket you have not sampled recently. The model is uncertain, but humans are not—that pair is a high-value probe, and the disagreement between model and humans tells you the reward model has a real gap. Now suppose the same pair shows high model uncertainty and scattered annotator disagreement. The model is not the only confused party. Before spending more labels, check whether the annotation instructions actually explain what the pair is asking annotators to weigh.

The record turns disagreement from a vague quality concern into a concrete next action: relabel, revise instructions, regenerate candidates, or review the objective.

Common Mistakes That Quietly Corrupt Preference Data

Four failure modes account for most corrupted preference datasets.

Collecting only from easy, agreeing pairs. The reward model becomes confident in the middle and blind at the boundaries. It scores typical responses well and fails exactly where the policy needs guidance.

Treating all disagreement as noise. Removing contested comparisons erases the boundary information the reward model needs. But treating all disagreement as signal is just as wrong. The diagnostic step—systematic versus scattered versus careless—determines whether disagreement reveals a value tradeoff or an annotation problem.

Letting candidate generation drift. If pairs differ on irrelevant dimensions—length, formatting, tone—the reward model learns those features instead of the behavior you care about. The model optimizes for the wrong thing because the data taught it the wrong thing.

Skipping coverage checks. Without periodic checks across edge cases and failure modes, you discover the reward model's blind spots only after the policy exploits them during optimization.

Each mistake shares a root cause: treating preference data as a quantity problem rather than a quality problem.

A Practical Collection Loop You Can Run

Here is the workflow I would run for a reward-learning project, in order.

Step 1: Define the behavior dimensions. Before writing any prompt or pair, name the behaviors you care about and the tradeoffs between them. Safety versus helpfulness. Brevity versus completeness. This definition drives every later decision.

Step 2: Generate candidates that span those dimensions. Deliberately include edge cases and failure modes, not just typical requests. The policy will encounter the boundaries during optimization; the data must cover them.

Step 3: Triage every candidate pair. Check relevance and validity first. Then use a provisional reward model to find pairs it cannot separate, and mix in coverage samples to prevent drift toward outliers.

Step 4: Annotate with clear priority instructions. Name the priority order explicitly. Use multiple annotators per comparison. Include an explicit tie or "both poor" option so annotators never have to fake a preference.

Step 5: Log disagreement and inspect its pattern. Systematic splits signal a contested objective. Scattered disagreement signals unclear instructions. Feed the diagnosis back into either better instructions or a sharper statement of the objective.

Each loop iteration should measurably reduce reward-model uncertainty on the boundary cases that matter. If it does not, the collection design is wrong—not the annotators.

The Decision Rule

Treat every annotation dollar as a measurement, not a label. Before collecting more data, ask which boundary the current reward model cannot separate, and design the next batch to probe exactly that boundary.

The reward model you end up with will only be as sharp as the comparisons that trained it. Disagreement logs are not a quality problem to clean up; they are the input that should drive your next audit of what the reward model actually learned. When you fit and audit that model, the questions you asked during collection—which boundary is uncertain, where do annotators split, what tradeoff is contested—are exactly the questions the audit should answer.