CLIP:用自然语言监督学习可迁移的视觉模型Learning Transferable Visual Models From Natural Language Supervision
问题:视觉模型通常预测固定类别,换一个概念就要重新标数据;能不能直接用互联网图文对做对比学习,让自然语言成为分类器?
证据位置:方法在 §2(对比学习、zero-shot 分类器构造);分析在 §3(细粒度、计数、OCR、分布偏移等);局限集中在 §6,作者把散落在各节的问题汇总成一份清单。
作者自陈局限:§6 写得非常直白——zero-shot CLIP 平均只是接近「ResNet-50 特征 + 线性分类器」这一简单基线,要达到整体 SOTA 估计还需要约 1000 倍算力,当前硬件不可行;在细粒度分类(车型、花种、飞机型号)、计数、以及训练分布外的新任务上很弱,MNIST 手写数字只有 88%,不如原始像素上的逻辑回归;只能从给定分类器里的概念中选,不能像 caption 模型那样生成新描述;没有解决数据效率问题;开发过程中反复在完整验证集上调参,不符合真正 zero-shot 的设定;评测集是「有点随意拼出来的 27 个数据集」,与 CLIP 能力共同演化;训练数据未筛选,会学到社会偏见。
它没主张的:没有说 zero-shot 已经全面超过监督模型——原文说「often competitive」;没有说 CLIP 能处理所有视觉概念——§6 明确说很多任务接近随机;也没有说 4 亿图文对是数据效率的解法——作者反而说它只是用规模换监督。Problem: vision models usually predict a fixed label set and need new labels for every new concept; can internet image-text pairs be used for contrastive pretraining so natural language itself becomes the classifier?
Where the evidence sits: approach in Section 2 (contrastive pretraining and zero-shot classifier construction); analysis in Section 3 (fine-grained, counting, OCR, distribution shift); limitations are collected in Section 6.
Stated limits: Section 6 is unusually direct — zero-shot CLIP is on average only competitive with a linear classifier on ResNet-50 features; reaching overall state of the art is estimated to need about 1000x more compute, infeasible on current hardware; it is weak on fine-grained classification (car models, flower species, aircraft variants), counting, and novel out-of-distribution tasks, scoring only 88% on MNIST handwritten digits, below logistic regression on raw pixels; it can only choose among concepts in a given classifier, unlike a caption model that can generate novel descriptions; it does not solve data efficiency; development repeatedly queried full validation sets, which is unrealistic for true zero-shot; the 27-dataset evaluation collection was somewhat haphazardly assembled and co-adapted with CLIP; and unfiltered training pairs teach social biases.
What it does not claim: it does not claim zero-shot beats supervised models across the board — the paper says “often competitive”; it does not claim CLIP handles all visual concepts — Section 6 says many tasks are near chance; and it does not claim 400M pairs solve data efficiency, arguing instead that CLIP trades scale for supervision.
关键参数Key specs
- 证据位置 / evidence
- §2 方法;§3 分析;§6 Limitations(汇总清单)
- 规模 / scale
- 4 亿(image, text)对;30+ 个视觉数据集
- 关键数字 / key numbers
- ImageNet zero-shot 追平原始 ResNet-50(未用其 128 万训练样本);MNIST 88%
- 自陈局限 / stated limits
- 需约 1000× 算力才到整体 SOTA;细粒度/计数/新任务弱;数据效率未解决;评测集共适应;社会偏见
- 复现状态 / reproduction
- 未复现
官方原文怎么说What the source says
we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on
we estimate around a 1000x increase in compute is required for zero-shot CLIP to reach overall state-of-the-art performance. This is infeasible to train with current hardware.
CLIP only achieves 88% accuracy on the handwritten digits of MNIST. An embarrassingly simple baseline of logistic regression on raw pixels outperforms zero-shot CLIP.
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-17
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2103.00020
- 类目Category
- 架构Architecture
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards