dorsal/arxiv
View SchemaPRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
| Authors | Ruiyi Ding, Yongxuan Lv, Xianhui Meng, Jiahe Song, Chao Wang, Chen Jiang, Yuan Cheng |
|---|---|
| Categories | |
| ArXiv ID | 2601.07182vv1 |
| URL | https://arxiv.org/abs/2601.07182 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization. Code is available at: https://github.com/SchumiDing/srpocode
{
"annotation_id": "f94c2b36-a683-406d-8dd7-8f25b20b1b94",
"date_created": "2026-02-17T05:53:11.752000Z",
"date_modified": "2026-02-17T05:53:11.752000Z",
"file_hash": "0c12ef422474590b519c5e01f78adaf88bf932a9eabff010b1234c6922109c23",
"private": false,
"record": {
"abstract": "Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization. Code is available at: https://github.com/SchumiDing/srpocode",
"arxiv_id": "2601.07182",
"authors": [
"Ruiyi Ding",
"Yongxuan Lv",
"Xianhui Meng",
"Jiahe Song",
"Chao Wang",
"Chen Jiang",
"Yuan Cheng"
],
"categories": [
"cs.LG",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization",
"url": "https://arxiv.org/abs/2601.07182",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "29916041-e04a-497f-a1f5-2909d1d1e75d",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}