跳到主要内容Skip to main content

← 返回AI 论文← Back to AI Papers

架构Architecture

Retrofitting language models to operate over bytes(Byteification)Retrofitting language models to operate over bytes (Byteification)

问题:主流 LLM 先把文本切成子词,字符级信息被遮住;代码、生物序列这类「单个字符/字节有意义」的数据尤其吃亏。字节级模型能避开分词,但此前性能一直落后。
证据位置:摘要给出方法与 49.1B token(不到典型预训练预算 1%)的转换成本;正文「Tokenizer transfer and retrofitting」说明两阶段流程;结果段报告 Bolmo 7B/1B、Bwen 8B、Blama 8B 等转换模型在字符级推理上超过早期字节级方法,并给出实际推理速度。
作者自陈局限:49.1B token 虽不到 1%,绝对量仍大;方法依赖一个已有子词模型作为教师,换源模型要重做;字节级推理需要专门内核,通用框架的支持并不自动成立;论文没有覆盖所有任务类型。
它没主张的:不是从零训练字节级模型;不是说字节级在所有任务上超过子词模型;不是宣布 tokenizer 无用;不是给出任意模型都能低成本转换的保证。
Problem: mainstream LLMs split text into subwords first, hiding character-level information; code and biological sequences, where individual characters or bytes matter, suffer most. Byte-level models avoid tokenisation but had lagged in performance.
Where the evidence sits: the abstract gives the method and the 49.1B-token conversion cost (less than 1% of a typical pretraining budget); the “Tokenizer transfer and retrofitting” section describes the two-stage procedure; the results report Bolmo 7B/1B, Bwen 8B and Blama 8B and character-level reasoning gains over earlier byte-level approaches, plus practical inference speed.
What the authors say about limits: 49.1B tokens is under 1% but still large in absolute terms; the method needs an existing subword model as teacher, so changing the source means redoing it; byte-level inference needs dedicated kernels, and generic framework support is not automatic; the paper does not cover every task type.
What it does not claim: training a byte-level model from scratch; that byte-level models beat subword models on every task; that tokenisers are obsolete; or that any model can be converted cheaply.

关键参数Key specs

发表 / venue
Nature,2026-10-07
DOI
10.1038/s41586-026-11111-4
转换成本 / budget
49.1B tokens(<1% 典型预训练预算)
报告模型 / models
Bolmo 7B/1B、Bwen 8B、Blama 8B
证据位置 / evidence
摘要;Tokenizer transfer and retrofitting;结果段
复现状态 / reproduction
未复现

官方原文怎么说What the source says

we introduce a general method for creating byte-level large language models through byteification that approach the capabilities of subword-based systems
1 https://doi.org/10.1038/s41586-026-11111-4
retrofitted an existing subword-level LLM to the byte level using less than 1% of a typical pretraining budget (49.1B tokens in total
2 https://doi.org/10.1038/s41586-026-11111-4
The resulting models outperform earlier byte-level approaches and excel on character-level reasoning tasks
3 https://doi.org/10.1038/s41586-026-11111-4
We use a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training.
4 https://doi.org/10.1038/s41586-026-11111-4

资料与许可Source and licence

资料时间Source checked
2026-11-18
许可 / 类型Licence / kind
journal-article
许可说明Licence evidence
https://doi.org/10.1038/s41586-026-11111-4
类目Category
架构Architecture

论文里的数字怎么读How to read the numbers

先分清测什么、怎么算、和谁比、有没有数据污染。拿不准就回原文看实验设置。Before citing a score, separate what it measures, how it is scored, what it compares against and whether contamination is controlled. If unsure, go back to the experiment setup in the paper. 看评测基准See benchmark cards · 看数据集卡See dataset cards

去官网看原页Open the official page →

← 返回AI 论文← Back to AI Papers