AgentBench: Evaluating LLMs as AgentsAgentBench: Evaluating LLMs as Agents
问题:LLM 作为 agent 在不同交互环境里的能力,缺一个统一、可比较的量尺。
证据位置:摘要写明 8 个环境,测推理与决策;商业模型与 ≤70B 开源模型差距明显;失败主因是长程推理、决策与指令跟随。
作者自陈局限:环境与任务是固定的文本交互,不是真实业务;商业 API 与开源模型的推理预算未统一;代码训练对不同 agent 任务影响矛盾,作者未给出统一因果。
它没主张的:不是真实业务代理排行榜;不是单一能力总分;不证明某个模型在所有环境都更强。Problem: there was no common yardstick for LLM-as-agent ability across interactive environments.
Where the evidence sits: the abstract states 8 distinct environments measuring reasoning and decision-making; it reports a significant gap between commercial models and OSS models up to 70B; the main failures are long-horizon reasoning, decision-making and instruction following.
What the authors say about limits: environments are fixed text interactions, not real business; inference budgets are not equalised between commercial APIs and OSS models; the effect of code training is ambivalent across agent tasks and no unified causal explanation is given.
What it does not claim: not a real-business agent leaderboard; not a single ability score; not proof that one model wins in every environment.
关键参数Key specs
- 证据位置 / evidence
- 摘要:8 个环境、≤70B 对比、失败原因
- 环境数 / environments
- 8
- 复现状态 / reproduction
- 未复现(只读摘要与原文关键段)
官方原文怎么说What the source says
a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities
there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B
poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-19
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2308.03688
- 类目Category
- Agent 与工具Agents & tools
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards