DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
问题:用 PPO 做数学推理强化学习时,显存与计算成本高;能否在不引入 critic 的情况下稳定优化。
证据位置:摘要提出 GRPO,作为 PPO 的变体,提升数学推理同时优化 PPO 的显存占用;DeepSeekMath 7B 在 MATH 得 51.7%(无工具、无投票);64 样本自一致性 60.9%。
作者自陈局限:数学域为主;7B 规模;依赖数据选择管线与奖励设计;通用对话/开放域未验证。
它没主张的:不是 PPO 的通用替代;不是所有任务都省显存;51.7% 不等于 GPT-4 水平。Problem: PPO for mathematical-reasoning RL is memory- and compute-heavy; can optimisation be stable without a critic?
Where the evidence sits: the abstract introduces GRPO as a PPO variant that improves mathematical reasoning while optimising PPO memory; DeepSeekMath 7B scores 51.7% on MATH without tools or voting, and 60.9% with 64-sample self-consistency.
What the authors say about limits: mainly the maths domain; 7B scale; depends on the data-selection pipeline and reward design; general chat/open domain is not validated.
What it does not claim: not a general replacement for PPO; not memory-saving on every task; 51.7% is not GPT-4 parity.
关键参数Key specs
- 证据位置 / evidence
- 摘要:GRPO、MATH 51.7%、64 样本 60.9%
- MATH / score
- 51.7%(7B,无工具/投票)
- 复现状态 / reproduction
- 未复现(只读摘要)
官方原文怎么说What the source says
we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO)
enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO
51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-19
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2402.03300
- 类目Category
- 对齐Alignment
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards