RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoFormer: Enhanced Transformer with Rotary Position Embedding
问题:Transformer 的位置编码如何同时表达绝对位置与相对位置依赖。
证据位置:摘要给出用旋转矩阵编码绝对位置,并在 self-attention 中显式引入相对位置依赖;具备序列长度灵活性、随相对距离衰减的 token 依赖;在长文本分类基准上稳定优于替代方法。
作者自陈局限:实验以长文本分类为主;理论分析有限;超出训练长度仍需调参;不同实现的数值稳定性有差异。
它没主张的:不是长度外推的完整解法;不是所有模型都必须用 RoPE;不是训练越长越好。Problem: how can Transformer positional encoding express both absolute position and relative-position dependency?
Where the evidence sits: the abstract encodes absolute position with a rotation matrix and injects explicit relative-position dependency into self-attention; it lists flexibility of sequence length and decaying inter-token dependency with distance; it reports consistent gains over alternatives on long-text classification benchmarks.
What the authors say about limits: experiments centre on long-text classification; theoretical analysis is limited; extending beyond training length still needs tuning; numerical stability varies across implementations.
What it does not claim: not a complete solution to length extrapolation; not mandatory for every model; not "longer training is always better".
关键参数Key specs
- 证据位置 / evidence
- 摘要:旋转矩阵、相对依赖、长文本分类
- 位置编码 / encoding
- Rotary Position Embedding
- 复现状态 / reproduction
- 未复现(只读摘要)
官方原文怎么说What the source says
encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention
flexibility of sequence length, decaying inter-token dependency with increasing relative distances
consistently overcomes its alternatives
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-19
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2104.09864
- 类目Category
- 架构Architecture
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards