Reinforcement Learning for LLMs: What the RL Loop Looks Like

If you have worked through policy gradients and deep RL, the language-model post-training landscape looks like a different discipline. RLHF. PPO. DPO. GRPO. RLAIF. A new acronym appears every quarter, each one presented as the thing that finally made models behave. The labels make the field feel like a collection of exotic algorithms invented in isolation.

They are not. Underneath the acronyms is the same agent-environment loop you already know. The policy is a language model. The action is choosing a token. The environment is whatever scores the text. Everything that looks exotic in LLM reinforcement learning is an implementation choice bolted onto that durable structure.

The Same Loop, a Different Action Space

The recurring confusion is that LLM post-training acronyms sound like a separate science. They are variations on one loop: sample behavior, score it, update the policy, constrain the update. You have seen this loop in deep RL. The only real difference is what occupies each role.

In the classic setting, the policy maps states to actions in a simulator or physical world. In the LLM setting, the policy is the language model itself. An action is choosing the next token from the vocabulary distribution. A trajectory is the generated sequence from prompt to stop token. The environment is the context plus whatever consumes the output.

That last piece is the unusual part. The environment is not physics. It is a judge. Someone—a human, a reward model, another LLM—reads the generated text and assigns it a score. That single difference drives most of what looks like a new discipline.

You already understand parameterized policies and policy gradients, so I will not re-teach them. What matters here is where the LLM setting bends those ideas: the action space is enormous, the trajectory is a text sequence, and the reward is constructed rather than given by the world.

States, Actions, and Trajectories at Three Granularities

To reason about any post-training paper, you need precise definitions of the RL vocabulary in the LLM setting. The terminology drifts across papers, but the underlying objects stay stable. The trick is that the same generation can be sliced at three different granularities, and papers often blur between them.

Token step. At the finest granularity, the state is the full conversation context: the prompt plus all tokens generated so far. The action is choosing the next token from the vocabulary distribution. The transition is deterministic—append the chosen token to the context. This is the natural granularity because the model is trained to produce a distribution over tokens at every position.

Response rollout. At the coarsest granularity, the whole response is treated as one sampled object. The state is the prompt, the action is the complete generated text, and the trajectory is a single step. This is not a contradiction of the token-level view. It is an abstraction choice: you are collapsing the internal token sequence because your reward signal only arrives at the end.

Interactive step. Between those two sits the setting that actually justifies RL machinery: multi-turn interaction and tool use. Here the model emits text, the environment responds with a tool result or a user message, and that new observation changes what the model can do next. The trajectory is a genuine sequence of environment steps, not just tokens inside one response.

Why does this distinction matter? Because it determines where credit assignment happens. Token-level optimization can assign credit to individual choices within a response. Response-level optimization treats the whole generation as one decision and scores it as a unit. Interactive optimization must assign credit across environment steps, where the reward for an early tool call only arrives after several subsequent actions.

There is a partial-observation nuance worth naming at every granularity. The model sees only its context window. It does not see the judge's criteria, the reward model's internal weights, or the human preferences that shaped them. The model acts on partial information about what will be scored well.

Where the Reward Comes From

Token prediction has a natural training signal: next-token likelihood. It has no natural reward for "good output." That is the core problem RL solves in this setting. RL enters when the objective is not next-token accuracy but judged quality—helpfulness, harmlessness, reasoning correctness, alignment with user intent.

Because text has no built-in scalar reward, the field builds one. That construction choice is the real dividing line between method families.

The RLHF path. Human preference comparisons train a reward model. Humans see two or more responses, rank them, and those comparisons become training data for a model that scores a whole response. The reward model turns human judgment into a scalar signal that can drive the RL loop. This is reinforcement learning from human feedback: the reward signal originates from human preferences, distilled into a learned model.

The AI-judge path. RLAIF replaces human annotators with an AI model that supplies the reward signal. This scales the loop dramatically—no human labeling bottleneck—but inherits the judge's biases. If the judging model prefers sycophantic text, the policy will learn sycophancy.

The direct preference path. Direct preference optimization and similar methods sit outside the full RL loop. They fold preference learning into the objective directly, skipping the separate reward model and the online sampling loop. The policy is still anchored to a reference model, and the objective still prefers one response over another—but there is no environment interaction, no reward estimation, and no trajectory-level credit assignment in the PPO sense.

Keep the distinction crisp: the reward source is a design decision. The loop that consumes the reward is the durable structure. Direct preference methods share the vocabulary of RL—policy, reference model, preference signal—without running the same sampled environment-interaction algorithm.

The tradeoff is practical. A reward model gives you a reusable, tunable signal, but it adds a training stage and can drift as the policy changes. Direct methods are simpler and cheaper, but they couple preference learning to the policy update, which makes them harder to extend to multi-turn or tool-use settings where the reward depends on a sequence of interactions.

The Policy Update and the Drift Problem

Once a reward signal exists, the update is a policy-gradient-style step. Sample responses, score them, push the model's probability mass toward higher-scoring text. If you have seen policy gradients, this is familiar territory with one new obsession: keeping the updated model close to its reference behavior.

The drift problem is real. An unconstrained update can push the model into degenerate text that games the reward model. Repetition. Sycophancy. Reward hacking—text that scores well but fails the actual objective. The model discovers that flattering the judge works better than answering the question.

The field's answer is KL divergence used as a leash. The update penalizes divergence from a reference policy, trading reward optimization against staying fluent and in-distribution. The reference policy is usually the model before RL training, and the constraint keeps the updated model from wandering into text that maximizes reward but breaks language quality.

This is where the method families diverge in implementation. PPO runs a full on-policy loop with a learned reward model and an explicit KL constraint. Direct preference methods rewrite the objective so the constraint is implicit in the preference data. Both are answering the same question: how do I move the policy toward preferred text without breaking it?

The constraint is not a footnote. It is the difference between a model that improves and one that collapses into reward hacking. When you read a paper and the authors emphasize KL penalties or reference-policy anchoring, they are managing the central failure mode of LLM reinforcement learning.

A Decision Map, Not a Taxonomy

Here is the map that cuts through the acronym soup. The families discussed above make three choices:

1. Reward source: a human-trained reward model, an AI judge, or a direct preference objective.
2. Update style: a full RL loop with sampling and iterative updates, or a direct gradient step on a preference objective.
3. Constraint mechanism: an explicit KL penalty against a reference policy, or an implicit constraint baked into the objective.

Note: This map is not a complete taxonomy. GRPO and similar recent methods add group-relative scoring—comparing multiple sampled responses against each other rather than against an absolute reward scale—which changes how the reward is normalized but not the underlying loop structure. When you encounter a new label, place it by asking which of the three choices it modifies.

The durable RL structure—sample, score, update, constrain—is identical across the full-loop rows. What changes is which component is learned and how the constraint is enforced.

I would caution you against treating the latest label as a paradigm shift. Most new methods recombine the same components. A paper that introduces a new acronym is usually changing the reward source, the update rule, or the constraint mechanism—not inventing a new form of learning.

When LLM RL Is Worth the Machinery

The honest question is not "does RLHF work?" It is "when does the full RL loop earn its complexity?"

RL earns its cost when the objective is genuinely sequential and judged. Multi-turn reasoning, tool use, agentic behavior—settings where the model's own actions change what happens next, and the final judgment depends on a long chain of decisions. Credit assignment across many generated tokens is exactly what the RL loop is built to handle.

Apply a sequentiality test before committing to the machinery. Does each intermediate action change the next observation or the available actions? Is the objective evaluated over the whole trajectory, not just the final text? If the answer to both is yes, you have a real sequential decision problem. If the answer is no—if a multi-step-looking prompt is actually scored as a single response—then response-level optimization may serve you just as well.

It is overkill when a single-turn preference signal would do. If you need a model to prefer concise answers over verbose ones, direct optimization or even supervised fine-tuning on curated outputs may be simpler and more stable. You do not need the full machinery of sampling, scoring, and constrained updates for a judgment that a single response can capture.

Where it shines is long horizons and delayed judgment. A model that must call a tool, read the result, revise its plan, and produce a final answer has a credit-assignment problem that supervised fine-tuning cannot address. The reward arrives at the end, but the decisions that caused it are spread across the trajectory. That is the RL loop's home turf.

State the boundary plainly: the loop is a tool for optimizing judged sequential behavior, not a mandatory finishing step for every model.

The Mental Model That Survives the Next Paper

Your next move is not to memorize acronyms. It is to trace one concrete pipeline end to end. Pick a method—RLHF with PPO, or a direct preference approach—and name its reward source, its update mechanism, and its constraint. Walk the loop: sample responses, score them, update the policy, check how far it moved from the reference.

When the next post-training paper appears, run it through the same map. Where does the reward come from? How is the policy updated? What keeps it from drifting into reward hacking? Answer those three questions and the acronym becomes a label on a familiar loop, not a magic spell.

The labels will keep changing. The loop will not.