2026-10-07Nature / 架构Nature / architecture
Byteification:把子词模型改成字节级Byteification: retrofitting subword models to bytes
用不到典型预训练预算 1% 的 49.1B token,把一个已有子词 LLM 转成字节级;报告 Bolmo 7B/1B、Bwen 8B、Blama 8B 在字符级推理上超过早期字节级方法。Converts an existing subword LLM to byte level with 49.1B tokens, less than 1% of a typical pretraining budget; reports Bolmo 7B/1B, Bwen 8B and Blama 8B beating earlier byte-level approaches on character-level reasoning.
- DOIDOI
- 10.1038/s41586-026-11111-4
- 预算budget
- 49.1B tokens(<1%)
- 模型models
- Bolmo 7B/1B、Bwen 8B、Blama 8B
- 复现reproduction
- 未复现
证据位置Where the evidence sits摘要给出方法与 49.1B token 的转换成本;正文「Tokenizer transfer and retrofitting」说明两阶段流程;结果段报告字符级任务与实际推理速度。The abstract gives the method and the 49.1B-token conversion cost; the “Tokenizer transfer and retrofitting” section describes the two-stage procedure; the results report character-level tasks and practical inference speed.
边界 / 它没主张什么Boundary / what it does not claim49.1B 虽不到 1%,绝对量仍大;方法依赖已有子词模型当教师;字节级推理需要专门内核;论文不覆盖所有任务。它没主张「字节级全面超过子词」或「tokenizer 无用」。49.1B is under 1% but still large in absolute terms; the method needs an existing subword teacher; byte-level inference needs dedicated kernels; the paper does not cover every task. It does not claim byte-level beats subword everywhere, or that tokenisers are obsolete.
来源Sources:Nature / DOINature / DOI
2026-10-29arXiv / 表征arXiv / representation
JEPA-PRH:109M 文本 JEPA 的跨模态对齐JEPA-PRH: cross-modal alignment from a 109M text JEPA
一个 109M 参数的文本 JEPA,在跨模态对齐上优于作者评测过的所有语言模型(最大到 72B);「好视角」比 next-token 少 42 倍训练样本。A 109M-parameter text JEPA aligns with image encoders better than every language model the authors evaluated, up to 72B; “good views” need 42 times fewer training samples than next-token prediction.
- arXivarXiv
- 2610.39302
- 作者authors
- Balestriero & LeCun
- 模型model
- 109M text JEPA
- 复现reproduction
- 未复现
证据位置Where the evidence sits摘要报告 109M JEPA 与 DINOv2 / CLIP 的对齐结果,以及 42× 样本效率;正文用 JEPA 作为可证明/可改进的实例。The abstract reports alignment with DINOv2 / CLIP and the 42× sample efficiency; the body uses JEPA as a provable/improveable instance.
边界 / 它没主张什么Boundary / what it does not claim预印本;109M 远小于被比较的 LLM;对齐指标不是生成质量;42× 是特定视角与数据条件,不是通用常数。它没主张「小模型全面超过大模型」或「PRH 已被最终证明」。A preprint; 109M is far smaller than the compared LLMs; alignment is not generation quality; 42× is specific to views and data, not a universal constant. It does not claim small models beat large models across the board, or that PRH is finally proved.
来源Sources:arXivarXiv
2026EMNLP / 推理EMNLP / reasoning
识别—拒答错位:知道不可答,却还是答了Recognition–refusal misalignment: knowing it is unanswerable, yet answering
论文在隐藏状态里找到一条线性方向,能把可答与「结构上不可能」的数学/代码题分开;识别方向与安全拒答方向接近正交,因此把失败解释为路由问题,而不是编码问题。The paper finds a linear direction in hidden states separating answerable from structurally impossible math/code prompts; recognition is nearly orthogonal to safety refusal, so the failure is a routing problem, not an encoding problem.
- 会议venue
- EMNLP 2026 Main
- ACL 编号ACL 编号
- 2026.emnlp-main.1078
- 任务tasks
- 数学 / 代码的结构性不可答
- 复现reproduction
- 未复现
证据位置Where the evidence sits线性探针能分开两类题;识别方向与拒答方向近正交。A linear probe separates the two classes; the recognition and refusal directions are nearly orthogonal.
边界 / 它没主张什么Boundary / what it does not claim只覆盖数学/代码的结构性不可答,不是所有「不知道」;线性可分不等于生成时一定使用该方向;未给通用修复方案。它没主张「所有幻觉都是路由问题」或「加一个拒答头就能解决」。It covers structurally unanswerable math/code, not every “don’t know”; linear separability does not prove the model uses that direction when generating; no general fix is offered. It does not claim every hallucination is routing, or that a refusal head solves it.
来源Sources:ACL AnthologyACL Anthology
2026ACL / 对齐ACL / alignment
Abstain-R1:3B 模型的可校准弃答Abstain-R1: calibrated abstention at 3B
用 clarification-aware RLVR 奖励训练 3B 模型,在不可答查询上同时改善弃答与澄清,并保持可答查询上的表现。Trains a 3B model with a clarification-aware RLVR reward, improving abstention and clarification on unanswerable queries while preserving answerable-query performance.
- arXivarXiv
- 2604.17073
- 会议venue
- ACL 2026
- 规模scale
- 3B
- 复现reproduction
- 未复现
证据位置Where the evidence sits摘要报告 3B 模型在不可答查询上的弃答/澄清提升,以及可答查询的性能保持;方法为可验证奖励 RL。The abstract reports gains in abstention/clarification on unanswerable queries and preserved performance on answerable ones; the method is verifiable-reward RL.
边界 / 它没主张什么Boundary / what it does not claim3B 规模,不能外推到 70B+;任务以不可答查询为主,开放域多轮澄清覆盖有限;奖励依赖可验证标签,标签有噪声;无生产验证。它没主张「弃答越多越好」或「解决所有幻觉」。3B scale, not extrapolable to 70B+; tasks centre on unanswerable queries, with limited open-domain multi-turn coverage; rewards depend on verifiable labels that carry noise; no production validation. It does not claim more abstention is always better, or that all hallucinations are solved.
来源Sources:arXivarXiv