RLHF: Giving the Shoggoth a Treat

The model was a chaotic blob, then we handed it a cookie, and now it says “Certainly!” before doing something wrong with confidence. Progress.

The reveal
Reinforcement Learning from Human Feedback (RLHF) is a practical recipe: train a model, collect human preferences over outputs, train a reward model, then optimize the model against that reward. It's one of the main reasons modern assistants feel less like autocomplete and more like tools. It's also the reason some people confuse "polite" with "aligned."

The steelman (yes, but also no)
Yes: RLHF can reduce obvious harmful outputs, improve helpfulness, and steer behavior toward user intent in many settings. No: RLHF does not guarantee truthfulness, robustness, or faithful goal alignment under distribution shift. It is a training signal, not a moral conversion.

What RLHF is actually doing (in plain mechanism terms)

So you've got three moving parts here. First, human preference labels—these optimize for "outputs humans like" under whatever labeling setup you've rigged, but they don't guarantee "outputs that are correct" when your model hits the wild. Then there's the reward model, which learns a proxy for preference but offers zero resistance to reward hacking. Finally, policy optimization cranks up the reward under that proxy, but stable behavior outside the proxy's coverage? Good luck with that.

If the proxy is your north star, don't act surprised when you end up in Proxy Alaska.

Receipts (the core papers, not vibes)

Let's start with Christiano et al.'s "Deep reinforcement learning from human preferences" from 2017—this was the early, clear demonstration that preference-based training could actually work as a scalable oversight approach. Then you've got Ouyang et al.'s InstructGPT paper from 2022, which showed RLHF producing preferred behavior even with smaller models, though they were honest about the limitations around value judgments and labeler effects. Bai et al. dropped "Constitutional AI" that same year, replacing some human labels with principles and self-critique—a neat illustration of both the promise and the "models supervise models" direction we're heading. More recently, OpenAI's Preparedness Framework v2 from 2025 laid out a concrete "capability thresholds lead to mitigations" approach that goes beyond just training models to be nicer. And if you want the reality check, the UK AI Security Institute's Frontier AI Trends Report shows how evaluation programs actually track risky capabilities, plus where the evidence gets mixed.

The limits, without drama
RLHF is strongest when the evaluation environment resembles deployment and when humans can reliably label "better" vs "worse." It weakens when tasks are long-horizon, adversarial, or require expertise humans don't have. It also pushes models toward saying the socially-approved thing, which can look like alignment until you test for truthfulness and resilience.

The pun mandate writes itself: RLHF is like slapping a "be nice" regularizer on a loss function and hoping the rest of the generalization does not gradient descend into weirdness.

Disproof conditions (what would make RLHF feel like 'alignment,' not 'polish')

Here's what would actually change our minds: If we saw strong robustness under distribution shift with minimal safety regression, we'd update because the "works in evals, fails in production" problem would finally shrink. If we got clear evidence that models resist reward hacking and deception across realistic adversarial tests, we'd update because the proxy would stop being trivially gameable. And if we witnessed scalable oversight that remains effective as models become more autonomous, we'd update because alignment would start to look like an engineering discipline, not a vibe.

Action line
Treat RLHF as part of a stack: evaluations, mitigations, governance, monitoring, and credible incident response. RLHF is the treat. It is not the leash, the fence, or the park ranger.

Anyway, back to touching grass—where reinforcement learning is mostly called "training a dog."

Related reading

Next up, check out "Alignment: This One Weird Trick" because the "one trick" is always "more work." Then hit "Vibes vs Evals" because preference is not measurement. And don't miss "Paperclip Maximizer and Other 80s B-Movies" because objective mismatch doesn't go away when the UI gets nicer.