Visual Instruction TuningVisual Instruction Tuning
问题:多模态指令跟随数据稀缺,能否用纯语言 GPT-4 生成图文指令数据,再训练端到端多模态模型。
证据位置:摘要称首次用 language-only GPT-4 生成多模态指令数据;LLaVA 连接视觉编码器与 LLM;在合成多模态指令集上相对 GPT-4 得 85.1%;ScienceQA 微调后 92.53%。
作者自陈局限:数据由 GPT-4 生成,继承其偏差;视觉编码器与 LLM 组合固定;评测以合成指令集与 ScienceQA 为主;真实场景鲁棒性未证明。
它没主张的:不是通用多模态助手;不是视觉理解全能;85.1% 不是全面超过 GPT-4。Problem: multimodal instruction-following data is scarce; can language-only GPT-4 generate image-text instruction data to train an end-to-end multimodal model?
Where the evidence sits: the abstract claims the first attempt to use language-only GPT-4 for multimodal instruction data; LLaVA connects a vision encoder and an LLM; it reports 85.1% relative score versus GPT-4 on a synthetic multimodal instruction set and 92.53% on ScienceQA after fine-tuning.
What the authors say about limits: data is GPT-4-generated and inherits its biases; the vision encoder and LLM pairing is fixed; evaluation centres on a synthetic instruction set and ScienceQA; real-world robustness is not established.
What it does not claim: not a general multimodal assistant; not all of visual understanding; 85.1% is not beating GPT-4 across the board.
关键参数Key specs
- 证据位置 / evidence
- 摘要:GPT-4 生成数据、85.1%、ScienceQA 92.53%
- 相对分数 / relative
- 85.1% vs GPT-4
- 复现状态 / reproduction
- 未复现(只读摘要)
官方原文怎么说What the source says
the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data
connects a vision encoder and LLM for general-purpose visual and language understanding
yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset
资料与许可Source and licence
- 资料时间Source checked
- 2026-11-19
- 许可 / 类型Licence / kind
- arxiv-preprint
- 许可说明Licence evidence
- https://arxiv.org/abs/2304.08485
- 类目Category
- 多模态Multimodal
论文里的数字怎么读How to read the numbers
先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards