dorsal/arxiv
View SchemaUserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning
| Authors | Feng Zhang, Shijia Li, Chunmao Zhang, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Jingwen Xu, Han Liu |
|---|---|
| Categories | |
| ArXiv ID | 2601.09215vv1 |
| URL | https://arxiv.org/abs/2601.09215 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set.
{
"annotation_id": "be689acf-29eb-4c64-a2f8-cbfdf6ae42b7",
"date_created": "2026-02-17T05:53:20.213000Z",
"date_modified": "2026-02-17T05:53:20.213000Z",
"file_hash": "93bf7bcd8d19f1edb3f4d22a4021ae63182a13de11f8b9e40b663e3c2ae6730c",
"private": false,
"record": {
"abstract": "User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set.",
"arxiv_id": "2601.09215",
"authors": [
"Feng Zhang",
"Shijia Li",
"Chunmao Zhang",
"Zhanyu Ma",
"Jun Xu",
"Jiuchong Gao",
"Jinghua Hao",
"Renqing He",
"Jingwen Xu",
"Han Liu"
],
"categories": [
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning",
"url": "https://arxiv.org/abs/2601.09215",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "82c90b4e-9e9c-4b9d-b53b-451d8f08218e",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}