Apple 机器学习 ↗ · 2026-11-12 08:00 · UTC+8
Apple 研究刻画语言模型对齐的理论极限
媒体报道 Apple 机器学习 原文语言:英文
Apple 研究人员从信息论角度刻画 KL 正则化对齐的极限,推导在 KL 预算下奖励提升的最大值。论文比较强化学习与 best-of-N 两类常见对齐方式,试图解释其根本限制。
全文
Theoretical Limits of Language Model Alignment
AuthorsLucas Monteiro Paes*, Natalie Mackraz*, Barry-John Theobald, Federico Danieli
Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NNN alignment, which selects the highest-reward output among NNN independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the √KL√KL√KL used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (reward hacking) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL–reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of-NNN closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.
* Equal contribution
正文以来源语言显示。