BERT:面向语言理解的深度双向 Transformer 预训练BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
问题:此前的语言模型要么是单向的(GPT 式从左到右),要么是两个单向模型拼接(ELMo 式),能不能在一个 Transformer 里同时看左右上下文做预训练?
证据位置:预训练目标在 §3.1(MLM 与 NSP),微调在 §3.2;主结果在 §4 与 Table 1–4;消融在 §5,其中 §5.1 的 Table 5 比较「No NSP」与「LTR & No NSP」,§5.2 的 Table 8 比较不同掩码比例;结论在 §6。
作者自陈:论文没有独立的 Limitations 章节,也没有系统列出局限。§5.1 的消融显示去掉 NSP 会显著伤 QNLI、MNLI 与 SQuAD 1.1;把模型改成从左到右会全面变差,MRPC 与 SQuAD 掉得最多。§5.2 说明微调对掩码策略相当稳健,但纯 [MASK] 策略在特征式 NER 上会出问题。
它没主张的:没有说双向一定优于所有双向变体(只比较了少数消融);没有说 NSP 本身不可替代(只是去掉会掉分);也没有给出多语言或长文档的通用结论——实验集中在英文 GLUE、MultiNLI 与 SQuAD。Problem: earlier language models were either unidirectional (GPT-style left-to-right) or a concatenation of two unidirectional models (ELMo-style); can one Transformer pretrain on both left and right context at once?
Where the evidence sits: pretraining objectives in 3.1 (MLM and NSP), fine-tuning in 3.2; main results in Section 4 and Tables 1-4; ablations in Section 5, with Table 5 in 5.1 comparing “No NSP” and “LTR & No NSP”, and Table 8 in 5.2 comparing masking rates; conclusion in Section 6.
What the authors say: there is no separate Limitations section and no systematic list. The 5.1 ablation shows removing NSP significantly hurts QNLI, MNLI and SQuAD 1.1, and switching to left-to-right degrades every task, with the largest drops on MRPC and SQuAD. Section 5.2 finds fine-tuning robust to masking strategies, while a pure [MASK] strategy is problematic for feature-based NER.
What it does not claim: it does not claim bidirectionality beats every bidirectional variant (only a few ablations were run), does not claim NSP is irreplaceable (only that removing it loses points), and gives no general multilingual or long-document conclusion — experiments centre on English GLUE, MultiNLI and SQuAD.
关键参数Key specs
- 证据位置 / evidence
- §3.1–3.2;§4 Table 1–4;§5.1 Table 5;§5.2 Table 8;§6
- 规模 / scale
- BERT-Base 与 BERT-Large 两档(论文 §3)
- 关键数字 / key numbers
- GLUE 80.5%(+7.7)、MultiNLI 86.7%(+4.6)、SQuAD v1.1 F1 93.2(+1.5)、SQuAD v2.0 F1 83.1(+5.1)
- 自陈局限 / stated limits
- 无独立 Limitations 章节;消融见 §5.1、§5.2
- 复现状态 / reproduction
- 未复现
官方原文怎么说What the source says
It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement)
removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1.
The LTR model performs worse than the MLM model on all tasks, with large drops on MRPC and SQuAD.
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-17
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/1810.04805
- 类目Category
- 架构Architecture
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards