ViT:一张图等于 16×16 个词An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
问题:Transformer 在 NLP 已经成为标准,但在视觉里通常还要和 CNN 绑定;能不能把图像切成 patch、当成词序列,直接用纯 Transformer 做分类?
证据位置:方法在 §3(patch 切分、线性投影、位置编码);实验在 §4,包含不同预训练数据规模(ImageNet、ImageNet-21k、JFT-300M)与 ResNet/hybrid 对比;结论在 §5。
作者自陈局限:§5 明确列出三项——还没把 ViT 用到检测与分割;自监督预训练有提升,但与大规模监督预训练仍有很大差距;继续扩大规模可能还会更好,但未验证。§4 也承认在小数据集上 ViT 比同算力 ResNet 更容易过拟合,卷积的归纳偏置在数据少时仍然有用。
它没主张的:没有说 ViT 在所有视觉任务上取代 CNN——结论把检测/分割列为未来工作;没有说自监督已经追平监督;也没有说 patch 16×16 是最优切法(做了 patch size 与位置编码消融,但结论是「规模优先于归纳偏置」这一范围内)。Problem: Transformers were already standard in NLP but usually tied to CNNs in vision; can an image be split into patches, treated as a token sequence, and classified by a pure Transformer?
Where the evidence sits: method in Section 3 (patch extraction, linear projection, positional encoding); experiments in Section 4 across pretraining data scales (ImageNet, ImageNet-21k, JFT-300M) and against ResNets and hybrids; conclusion in Section 5.
Stated limits: Section 5 lists three — ViT has not yet been applied to detection and segmentation; self-supervised pretraining helps but still lags large-scale supervised pretraining by a large gap; further scaling would likely help but was not verified. Section 4 also concedes that on small datasets ViT overfits more than a ResNet of comparable compute, so convolutional inductive bias still matters when data is scarce.
What it does not claim: it does not claim ViT replaces CNNs across all vision tasks — detection and segmentation are left to future work; it does not claim self-supervision has caught up with supervision; and it does not claim 16x16 is the optimal patch size beyond the ablations it ran.
关键参数Key specs
- 证据位置 / evidence
- §3 方法;§4 实验与数据规模消融;§5 结论与局限
- 规模 / scale
- ViT-Base / Large / Huge,patch 16 与 32 等变体
- 关键数字 / key numbers
- ImageNet 88.55%、ReaL 90.72%、CIFAR-100 94.55%、VTAB 19 任务 77.63%
- 自陈局限 / stated limits
- 检测/分割未验证;自监督仍落后;小数据易过拟合(§4、§5)
- 复现状态 / reproduction
- 未复现
官方原文怎么说What the source says
the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks
One is to apply ViT to other computer vision tasks, such as detection and segmentation.
Our initial experiments show improvement from self-supervised pre-training, but there is still large gap between self-supervised and large-scale supervised pre-training.
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-17
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2010.11929
- 类目Category
- 架构Architecture
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards