跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

多模态Multimodal

Visual Instruction TuningVisual Instruction Tuning

问题:多模态指令跟随数据稀缺,能否用纯语言 GPT-4 生成图文指令数据,再训练端到端多模态模型。
证据位置:摘要称首次用 language-only GPT-4 生成多模态指令数据;LLaVA 连接视觉编码器与 LLM;在合成多模态指令集上相对 GPT-4 得 85.1%;ScienceQA 微调后 92.53%。
作者自陈局限:数据由 GPT-4 生成,继承其偏差;视觉编码器与 LLM 组合固定;评测以合成指令集与 ScienceQA 为主;真实场景鲁棒性未证明。
它没主张的:不是通用多模态助手;不是视觉理解全能;85.1% 不是全面超过 GPT-4。
Problem: multimodal instruction-following data is scarce; can language-only GPT-4 generate image-text instruction data to train an end-to-end multimodal model?
Where the evidence sits: the abstract claims the first attempt to use language-only GPT-4 for multimodal instruction data; LLaVA connects a vision encoder and an LLM; it reports 85.1% relative score versus GPT-4 on a synthetic multimodal instruction set and 92.53% on ScienceQA after fine-tuning.
What the authors say about limits: data is GPT-4-generated and inherits its biases; the vision encoder and LLM pairing is fixed; evaluation centres on a synthetic instruction set and ScienceQA; real-world robustness is not established.
What it does not claim: not a general multimodal assistant; not all of visual understanding; 85.1% is not beating GPT-4 across the board.

关键参数Key specs

证据位置 / evidence
摘要:GPT-4 生成数据、85.1%、ScienceQA 92.53%
相对分数 / relative
85.1% vs GPT-4
复现状态 / reproduction
未复现(只读摘要)

官方原文怎么说What the source says

the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data
1 https://arxiv.org/abs/2304.08485
connects a vision encoder and LLM for general-purpose visual and language understanding
2 https://arxiv.org/abs/2304.08485
yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset
3 https://arxiv.org/abs/2304.08485

资料与许可Source and licence

资料时间Source checked
2026-11-19
许可 / 类型Licence / kind
arxiv-preprint
许可说明Licence evidence
https://arxiv.org/abs/2304.08485
类目Category
多模态Multimodal

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers