跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

效率Efficiency

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersGPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

问题:几百亿参数的模型光显存就装不下,训练后量化到低比特往往掉点严重;能不能在不重训的前提下压到 3–4 bit 还基本不掉。
证据位置:175B 模型约 4 个 GPU 小时压到 3 或 4 bit 的结论在摘要;局限同时写在引言贡献段与第 6 节 Summary and Limitations。
作者自陈局限:两点很关键——加速来自减少显存搬运,不是减少计算量,主流硬件不支持 FP16×INT4 这类混合精度乘加,因此实际矩阵乘法并不会变快;研究只覆盖生成任务,未做激活量化(作者认为激活不是目标场景的瓶颈)。
它没主张的:论文没有说量化后推理整体更快,也没有覆盖激活量化与更大的压缩率;「单卡跑 175B」指的是权重能装进一张卡,不等于该配置的吞吐满足线上服务。
Problem: models with tens of billions of parameters do not fit in memory, and post-training quantisation to low bit-width usually costs accuracy; can 3-4 bits be reached without retraining and with almost no loss.
Where the evidence sits: the 175B-in-four-GPU-hours result at 3 or 4 bits is in the abstract; the limits appear both in the introduction’s contribution list and in Section 6, Summary and Limitations.
What the authors say about limits: two matter — the speed-up comes from reduced memory movement, not from fewer computations, and mainstream hardware lacks support for mixed-precision operands such as FP16 by INT4, so the actual multiplications are not faster; and the study covers generative tasks only and does not quantise activations, which the authors consider not a bottleneck in their target setting.
What it does not claim: the paper does not claim faster end-to-end inference, nor cover activation quantisation or higher compression; "175B on a single GPU" means the weights fit on one card, not that throughput meets serving requirements.

关键参数Key specs

证据位置 / evidence
摘要;引言贡献段;Section 6 Summary and Limitations
位宽 / bit-width
3 或 4 bit 每权重
量化耗时 / quantisation
175B 模型约 4 GPU 小时
加速来源 / speed-up source
减少显存搬运,非减少计算
未覆盖 / not covered
激活量化
复现状态 / reproduction
未复现

官方原文怎么说What the source says

Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline.
1 https://ar5iv.labs.arxiv.org/html/2210.17323
In terms of limitations, our method currently does not provide speedups for the actual multiplications, due to the lack of hardware support for mixed-precision operands (e.g. FP16 x INT4) on mainstream architectures.
2 https://ar5iv.labs.arxiv.org/html/2210.17323
On the technical side, our method obtains speedups from reduced memory movement, and does not lead to computational reductions.
3 https://ar5iv.labs.arxiv.org/html/2210.17323

资料与许可Source and licence

资料时间Source checked
2026-11-07
许可 / 类型Licence / kind
arxiv-preprint
许可说明Licence evidence
https://arxiv.org/abs/2210.17323
类目Category
效率Efficiency

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers