{
  "annotation_id": "6a038ea5-2a91-4afe-978d-7b692214edee",
  "date_created": "2026-02-17T05:53:16.294000Z",
  "date_modified": "2026-02-17T05:53:16.294000Z",
  "file_hash": "1fe0f93eeac6d6463d2af9c071923e25a095f0bfebbb1a71e33ca3c2d809993f",
  "private": false,
  "record": {
    "abstract": "Aligning large language models (LLMs) to serve users with heterogeneous and potentially conflicting preferences is a central challenge for personalized and trustworthy AI. We formalize an ideal notion of universal alignment through test-time scaling: for each prompt, the model produces $k\\ge 1$ candidate responses and a user selects their preferred one. We introduce $(k,f(k))$-robust alignment, which requires the $k$-output model to have win rate $f(k)$ against any other single-output model, and asymptotic universal alignment (U-alignment), which requires $f(k)\\to 1$ as $k\\to\\infty$. Our main result characterizes the optimal convergence rate: there exists a family of single-output policies whose $k$-sample product policies achieve U-alignment at rate $f(k)=\\frac{k}{k+1}$, and no method can achieve a faster rate in general.\n  We show that popular post-training methods, including Nash learning from human feedback (NLHF), can fundamentally underutilize the benefits of test-time scaling. Even though NLHF is optimal for $k=1$, sampling from the resulting (often deterministic) policy cannot guarantee win rates above $\\tfrac{1}{2}$ except for an arbitrarily small slack. This stems from a lack of output diversity: existing alignment methods can collapse to a single majority-preferred response, making additional samples redundant. In contrast, our approach preserves output diversity and achieves the optimal test-time scaling rate. In particular, we propose a family of symmetric multi-player alignment games and prove that any symmetric Nash equilibrium policy of the $(k+1)$-player alignment game achieves the optimal $(k,\\frac{k}{k+1})$-robust alignment. Finally, we provide theoretical convergence guarantees for self-play learning dynamics in these games and extend the framework to opponents that also generate multiple responses.",
    "arxiv_id": "2601.08777",
    "authors": [
      "Yang Cai",
      "Weiqiang Zheng"
    ],
    "categories": [
      "cs.LG",
      "cs.AI",
      "cs.CL",
      "cs.GT"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "title": "Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling",
    "url": "https://arxiv.org/abs/2601.08777",
    "version": "v1"
  },
  "schema_id": "dorsal/arxiv",
  "source": {
    "execution_id": "8e25d23b-c203-4475-99d0-92d4f2491d66",
    "id": "arXiv Dataset",
    "type": "Model",
    "variant": "snapshot-2026-01-17",
    "version": "0.1.0"
  },
  "user_id": 1000002
}