Switch Transformer:用简单稀疏路由扩到万亿参数Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
问题:MoE 参数多、算力恒定,但路由复杂、通信贵、训练不稳,所以一直没普及;能不能把路由简化成「每个 token 只选一个专家」,再配一套训练技巧把它跑稳?
证据位置:Switch 层与 top-1 路由在 §2;容量因子与负载均衡损失在 §2.2;训练与稳定性技巧(选择性精度、初始化、正则)在 §3;结果在 §4–§5,包含 T5-Base/Large 同算力 7× 预训练加速、101 种语言的 mT5 增益、万亿参数 Switch-C 对 T5-XXL 的 4× 加速;结论在 §9。
作者自陈局限:论文没有独立 Limitations 章节。Appendix E 明确提醒:上游困惑度不一定转化为下游任务——在最大模型区间(T5-11B/T5-XXL),Switch 模型并不总能把预训练优势带到 SuperGLUE 微调;作者说这需要未来研究,并指出专家模型的微调动态很复杂,依赖正则、负载均衡与超参。
它没主张的:没有说 Switch 在所有下游任务上都优于 dense——Appendix E 反而说小中模型区间两者相似,最大模型在 SuperGLUE 上不一定更好;没有说 top-1 路由普适最优——只与 GShard 等少数路由做了比较;也没有说 7× 加速能直接换算成 7× 成本优势——通信与专家并行开销未计入这句口号。Problem: MoE has many parameters at constant compute, but complex routing, expensive communication and unstable training kept it niche; can routing be simplified to one expert per token and paired with training tricks to make it stable?
Where the evidence sits: the Switch layer and top-1 routing in Section 2, capacity factor and load-balancing loss in 2.2, training and stability techniques (selective precision, initialization, regularization) in Section 3; results in Sections 4-5, including 7x pretraining speedup at equal compute for T5-Base/Large, gains across 101 languages for mT5, and a 4x speedup of trillion-parameter Switch-C over T5-XXL; conclusion in Section 9.
Stated limits: no separate Limitations section. Appendix E warns that upstream perplexity does not necessarily transfer downstream — in the largest regime (T5-11B/T5-XXL) Switch models do not always carry their pretraining advantage into SuperGLUE fine-tuning, which the authors leave to future work and attribute to complex fine-tuning dynamics depending on regularization, load balancing and hyperparameters.
What it does not claim: it does not claim Switch beats dense models on every downstream task — Appendix E says the two are similar in small-to-medium regimes and the largest Switch model is not always better on SuperGLUE; it does not claim top-1 routing is universally optimal, comparing only a few alternatives such as GShard; and it does not claim the 7x speedup translates directly into 7x cost savings, since communication and expert-parallel overhead are outside that headline.
关键参数Key specs
- 证据位置 / evidence
- §2 路由与负载均衡;§3 训练技巧;§4–5 结果;§9 结论;Appendix E 下游迁移
- 规模 / scale
- T5-Base/Large 至 1.6T 参数、2048 专家的 Switch-C
- 关键数字 / key numbers
- 同算力最高 7× 预训练加速;101 语言优于 mT5-Base;相对 T5-XXL 4× 加速
- 自陈局限 / stated limits
- 最大模型的上游困惑度不总能迁移到 SuperGLUE(Appendix E);无独立 Limitations 章节
- 复现状态 / reproduction
- 未复现
官方原文怎么说What the source says
we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats
obtain up to 7x increases in pre-training speed with the same computational resources
do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-17
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2101.03961
- 类目Category
- 规模律Scaling laws
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards