跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

对齐Alignment

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

问题:用 PPO 做数学推理强化学习时,显存与计算成本高;能否在不引入 critic 的情况下稳定优化。
证据位置:摘要提出 GRPO,作为 PPO 的变体,提升数学推理同时优化 PPO 的显存占用;DeepSeekMath 7B 在 MATH 得 51.7%(无工具、无投票);64 样本自一致性 60.9%。
作者自陈局限:数学域为主;7B 规模;依赖数据选择管线与奖励设计;通用对话/开放域未验证。
它没主张的:不是 PPO 的通用替代;不是所有任务都省显存;51.7% 不等于 GPT-4 水平。
Problem: PPO for mathematical-reasoning RL is memory- and compute-heavy; can optimisation be stable without a critic?
Where the evidence sits: the abstract introduces GRPO as a PPO variant that improves mathematical reasoning while optimising PPO memory; DeepSeekMath 7B scores 51.7% on MATH without tools or voting, and 60.9% with 64-sample self-consistency.
What the authors say about limits: mainly the maths domain; 7B scale; depends on the data-selection pipeline and reward design; general chat/open domain is not validated.
What it does not claim: not a general replacement for PPO; not memory-saving on every task; 51.7% is not GPT-4 parity.

关键参数Key specs

证据位置 / evidence
摘要:GRPO、MATH 51.7%、64 样本 60.9%
MATH / score
51.7%(7B,无工具/投票)
复现状态 / reproduction
未复现(只读摘要)

官方原文怎么说What the source says

we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO)
1 https://arxiv.org/abs/2402.03300
enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO
2 https://arxiv.org/abs/2402.03300
51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques
3 https://arxiv.org/abs/2402.03300

资料与许可Source and licence

资料时间Source checked
2026-11-19
许可 / 类型Licence / kind
arxiv-preprint
许可说明Licence evidence
https://arxiv.org/abs/2402.03300
类目Category
对齐Alignment

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers