跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

架构Architecture

RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoFormer: Enhanced Transformer with Rotary Position Embedding

问题:Transformer 的位置编码如何同时表达绝对位置与相对位置依赖。
证据位置:摘要给出用旋转矩阵编码绝对位置,并在 self-attention 中显式引入相对位置依赖;具备序列长度灵活性、随相对距离衰减的 token 依赖;在长文本分类基准上稳定优于替代方法。
作者自陈局限:实验以长文本分类为主;理论分析有限;超出训练长度仍需调参;不同实现的数值稳定性有差异。
它没主张的:不是长度外推的完整解法;不是所有模型都必须用 RoPE;不是训练越长越好。
Problem: how can Transformer positional encoding express both absolute position and relative-position dependency?
Where the evidence sits: the abstract encodes absolute position with a rotation matrix and injects explicit relative-position dependency into self-attention; it lists flexibility of sequence length and decaying inter-token dependency with distance; it reports consistent gains over alternatives on long-text classification benchmarks.
What the authors say about limits: experiments centre on long-text classification; theoretical analysis is limited; extending beyond training length still needs tuning; numerical stability varies across implementations.
What it does not claim: not a complete solution to length extrapolation; not mandatory for every model; not "longer training is always better".

关键参数Key specs

证据位置 / evidence
摘要:旋转矩阵、相对依赖、长文本分类
位置编码 / encoding
Rotary Position Embedding
复现状态 / reproduction
未复现(只读摘要)

官方原文怎么说What the source says

encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention
1 https://arxiv.org/abs/2104.09864
flexibility of sequence length, decaying inter-token dependency with increasing relative distances
2 https://arxiv.org/abs/2104.09864
consistently overcomes its alternatives
3 https://arxiv.org/abs/2104.09864

资料与许可Source and licence

资料时间Source checked
2026-11-19
许可 / 类型Licence / kind
arxiv-preprint
许可说明Licence evidence
https://arxiv.org/abs/2104.09864
类目Category
架构Architecture

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers