YaRN: Efficient Context Window Extension of Large Language ModelsYaRN: Efficient Context Window Extension of Large Language Models
问题:RoPE 模型超出训练长度后失效,如何用更少 token 与训练步数扩展上下文窗口。
证据位置:摘要称比此前方法少 10× token、少 2.5× 训练步;LLaMA 可外推到远长于预训练的长度;在上下文扩展上超过此前 SOTA;可外推超过微调数据集的上下文。
作者自陈局限:以 LLaMA 系列为主;需要少量微调;超长上下文仍受注意力成本限制;未证明所有任务保真。
它没主张的:不是零成本无限上下文;不是不需要微调;不是所有模型通用。Problem: RoPE models fail beyond their training length; how to extend the context window with fewer tokens and training steps?
Where the evidence sits: the abstract claims 10x fewer tokens and 2.5x fewer training steps than prior methods; LLaMA models extrapolate far beyond their original pretraining length; it surpasses prior SOTA on context extension and can extrapolate beyond the fine-tuning dataset context.
What the authors say about limits: mainly LLaMA-family experiments; requires light fine-tuning; long contexts still pay attention cost; fidelity on all tasks is not proven.
What it does not claim: not zero-cost infinite context; not fine-tuning-free; not universal across models.
关键参数Key specs
- 证据位置 / evidence
- 摘要:10× token、2.5× 步数、LLaMA 外推
- 训练成本 / training
- 10× fewer tokens; 2.5× fewer steps
- 复现状态 / reproduction
- 未复现(只读摘要)
官方原文怎么说What the source says
requiring 10x less tokens and 2.5x less training steps than previous methods
LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow
extrapolate beyond the limited context of a fine-tuning dataset
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-19
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2309.00071
- 类目Category
- 效率Efficiency
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards