dorsal/arxiv
View SchemaSegmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training
| Authors | Xue Gong, Qi Yi, Ziyuan Nan, Guanhua Huang, Kejiao Li, Yuhao Jiang, Ruibin Xiong, Zenan Xu, Jiaming Guo, Shaohui Peng, Bo Zhou |
|---|---|
| Categories | |
| ArXiv ID | 2601.07320vv1 |
| URL | https://arxiv.org/abs/2601.07320 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Training Large Language Models (LLMs) for reasoning tasks is increasingly driven by Reinforcement Learning with Verifiable Rewards (RLVR), where Proximal Policy Optimization (PPO) provides a principled framework for stable policy updates. However, the practical application of PPO is hindered by unreliable advantage estimation in the sparse-reward RLVR regime. This issue arises because the sparse rewards in RLVR lead to inaccurate intermediate value predictions, which in turn introduce significant bias when aggregated at every token by Generalized Advantage Estimation (GAE). To address this, we introduce Segmental Advantage Estimation (SAE), which mitigates the bias that GAE can incur in RLVR. Our key insight is that aggregating $n$-step advantages at every token(as in GAE) is unnecessary and often introduces excessive bias, since individual tokens carry minimal information. Instead, SAE first partitions the generated sequence into coherent sub-segments using low-probability tokens as heuristic boundaries. It then selectively computes variance-reduced advantage estimates only from these information-rich segment transitions, effectively filtering out noise from intermediate tokens. Our experiments demonstrate that SAE achieves superior performance, with marked improvements in final scores, training stability, and sample efficiency. These gains are shown to be consistent across multiple model sizes, and a correlation analysis confirms that our proposed advantage estimator achieves a higher correlation with an approximate ground-truth advantage, justifying its superior performance.
{
"annotation_id": "b27331f7-1f82-4431-81ec-a8a7b7b94237",
"date_created": "2026-02-17T05:53:12.580000Z",
"date_modified": "2026-02-17T05:53:12.580000Z",
"file_hash": "04931f489993bfca30feac7df21dbe3a4cd2b0fd6323fa2b69f0ea10eb5e76a1",
"private": false,
"record": {
"abstract": "Training Large Language Models (LLMs) for reasoning tasks is increasingly driven by Reinforcement Learning with Verifiable Rewards (RLVR), where Proximal Policy Optimization (PPO) provides a principled framework for stable policy updates. However, the practical application of PPO is hindered by unreliable advantage estimation in the sparse-reward RLVR regime. This issue arises because the sparse rewards in RLVR lead to inaccurate intermediate value predictions, which in turn introduce significant bias when aggregated at every token by Generalized Advantage Estimation (GAE). To address this, we introduce Segmental Advantage Estimation (SAE), which mitigates the bias that GAE can incur in RLVR. Our key insight is that aggregating $n$-step advantages at every token(as in GAE) is unnecessary and often introduces excessive bias, since individual tokens carry minimal information. Instead, SAE first partitions the generated sequence into coherent sub-segments using low-probability tokens as heuristic boundaries. It then selectively computes variance-reduced advantage estimates only from these information-rich segment transitions, effectively filtering out noise from intermediate tokens. Our experiments demonstrate that SAE achieves superior performance, with marked improvements in final scores, training stability, and sample efficiency. These gains are shown to be consistent across multiple model sizes, and a correlation analysis confirms that our proposed advantage estimator achieves a higher correlation with an approximate ground-truth advantage, justifying its superior performance.",
"arxiv_id": "2601.07320",
"authors": [
"Xue Gong",
"Qi Yi",
"Ziyuan Nan",
"Guanhua Huang",
"Kejiao Li",
"Yuhao Jiang",
"Ruibin Xiong",
"Zenan Xu",
"Jiaming Guo",
"Shaohui Peng",
"Bo Zhou"
],
"categories": [
"cs.LG",
"cs.AI"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training",
"url": "https://arxiv.org/abs/2601.07320",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "4ca4c872-34a5-43dd-b090-17fa23c30bd8",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}