跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

对齐Alignment

Training language models to follow instructions with human feedbackTraining language models to follow instructions with human feedback

问题:让模型跟着用户意图走,而不只是把下一词预测得更准;做法是用人类偏好数据做监督微调再训奖励模型做强化学习。
证据位置:摘要给出 1.3B 胜过 175B 的偏好结论;局限集中在第 5.3 节 Limitations,对齐对象的讨论在第 5.2 节,未解决问题在第 5.4 节。
作者自陈局限:写了三层——标注者以英语为主、数据几乎全是英文指令,不能代表全部使用者;模型既不完全对齐也不完全安全,仍会编造事实、生成有毒或偏见内容;作者认为最大的局限是模型多数情况下会服从用户指令,即使这在现实世界可能造成伤害。他们还明确说并不主张研究者、雇来的标注者或 API 客户就是偏好的正确来源。
它没主张的:论文没有说对齐问题已被解决,也没有给出一个能同时满足所有人偏好的方案——作者原话是不可能训练出同时对齐所有人偏好、或让所有人都认可其取舍的系统。
Problem: make a model follow user intent rather than only predict the next token better, using human preference data for supervised fine-tuning followed by reward-model-driven reinforcement learning.
Where the evidence sits: the abstract gives the 1.3B-beats-175B preference result; the limits are concentrated in Section 5.3 (Limitations), the question of whose preferences count is in 5.2, and open questions in 5.4.
What the authors say about limits: three layers — labelers are primarily English-speaking and the data is almost entirely English instructions, so they are not representative of everyone affected; the models are neither fully aligned nor fully safe and still fabricate facts or produce toxic and biased content; and the stated greatest limitation is that the models follow the user’s instruction even when that could cause real-world harm. They also state plainly that they are not claiming researchers, hired labelers or API customers are the right source of preferences.
What it does not claim: the paper does not say alignment is solved, nor offer a scheme that satisfies everyone — the authors write that it is impossible to train a system aligned to everyone’s preferences at once, or where everyone would endorse the tradeoffs.

关键参数Key specs

证据位置 / evidence
摘要;Section 5.3 Limitations;5.2 对齐对象;5.4 开放问题
方法 / method
SFT → 奖励模型 → PPO
摘要口径 / headline
1.3B 的输出被偏好于 175B GPT-3,参数量少 100 倍
复现状态 / reproduction
未复现

官方原文怎么说What the source says

Making language models bigger does not inherently make them better at following a user’s intent.
1 https://ar5iv.labs.arxiv.org/html/2203.02155
Our models are neither fully aligned nor fully safe; they still generate toxic or biased outputs, make up facts, and generate sexual and violent content without explicit prompting.
2 https://ar5iv.labs.arxiv.org/html/2203.02155
In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.
3 https://ar5iv.labs.arxiv.org/html/2203.02155

资料与许可Source and licence

资料时间Source checked
2026-11-07
许可 / 类型Licence / kind
arxiv-preprint
许可说明Licence evidence
https://arxiv.org/abs/2203.02155
类目Category
对齐Alignment

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers