Apple researchers characterize theoretical limits of language model alignment

Media reports Apple 机器学习 Source language: English

Apple researchers derive information-theoretic limits of KL-regularized alignment, including the maximum reward improvement under a KL budget. The paper compares reinforcement learning and best-of-N alignment to explain their fundamental limits.

Full text

Theoretical Limits of Language Model Alignment

AuthorsLucas Monteiro Paes*, Natalie Mackraz*, Barry-John Theobald, Federico Danieli

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NNN alignment, which selects the highest-reward output among NNN independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the √KL√KL√KL used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (reward hacking) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL–reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of-NNN closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.

* Equal contribution

Sources

← Back to Daily AI Report