dorsal/arxiv
View SchemaRewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
| Authors | Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi |
|---|---|
| Categories | |
| ArXiv ID | 2601.08763vv2 |
| URL | https://arxiv.org/abs/2601.08763 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.
{
"annotation_id": "932ec299-28e4-4185-9e6d-74a899aa24aa",
"date_created": "2026-02-17T05:53:16.074000Z",
"date_modified": "2026-02-17T05:53:16.074000Z",
"file_hash": "bcb0471b10d625b1f057016fc6cb97250157fe30b5cdef326ecd0b7acc581951",
"private": false,
"record": {
"abstract": "Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.",
"arxiv_id": "2601.08763",
"authors": [
"Zhiyuan Hu",
"Yucheng Wang",
"Yufei He",
"Jiaying Wu",
"Yilun Zhao",
"See-Kiong Ng",
"Cynthia Breazeal",
"Anh Tuan Luu",
"Hae Won Park",
"Bryan Hooi"
],
"categories": [
"cs.LG",
"cs.CL"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs",
"url": "https://arxiv.org/abs/2601.08763",
"version": "v2"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "38de641a-9ce2-4151-be57-f1c4460bf8cb",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}