跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

效率Efficiency

YaRN: Efficient Context Window Extension of Large Language ModelsYaRN: Efficient Context Window Extension of Large Language Models

问题:RoPE 模型超出训练长度后失效,如何用更少 token 与训练步数扩展上下文窗口。
证据位置:摘要称比此前方法少 10× token、少 2.5× 训练步;LLaMA 可外推到远长于预训练的长度;在上下文扩展上超过此前 SOTA;可外推超过微调数据集的上下文。
作者自陈局限:以 LLaMA 系列为主;需要少量微调;超长上下文仍受注意力成本限制;未证明所有任务保真。
它没主张的:不是零成本无限上下文;不是不需要微调;不是所有模型通用。
Problem: RoPE models fail beyond their training length; how to extend the context window with fewer tokens and training steps?
Where the evidence sits: the abstract claims 10x fewer tokens and 2.5x fewer training steps than prior methods; LLaMA models extrapolate far beyond their original pretraining length; it surpasses prior SOTA on context extension and can extrapolate beyond the fine-tuning dataset context.
What the authors say about limits: mainly LLaMA-family experiments; requires light fine-tuning; long contexts still pay attention cost; fidelity on all tasks is not proven.
What it does not claim: not zero-cost infinite context; not fine-tuning-free; not universal across models.

关键参数Key specs

证据位置 / evidence
摘要:10× token、2.5× 步数、LLaMA 外推
训练成本 / training
10× fewer tokens; 2.5× fewer steps
复现状态 / reproduction
未复现(只读摘要)

官方原文怎么说What the source says

requiring 10x less tokens and 2.5x less training steps than previous methods
1 https://arxiv.org/abs/2309.00071
LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow
2 https://arxiv.org/abs/2309.00071
extrapolate beyond the limited context of a fine-tuning dataset
3 https://arxiv.org/abs/2309.00071

资料与许可Source and licence

资料时间Source checked
2026-11-19
许可 / 类型Licence / kind
arxiv-preprint
许可说明Licence evidence
https://arxiv.org/abs/2309.00071
类目Category
效率Efficiency

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers