dorsal/arxiv
View SchemaSparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
| Authors | Sijia Luo, Xiaokang Zhang, Yuxuan Hu, Bohan Zhang, Ke Wang, Jinbo Su, Mengshu Sun, Lei Liang, Jing Zhang |
|---|---|
| Categories | |
| ArXiv ID | 2601.10079vv1 |
| URL | https://arxiv.org/abs/2601.10079 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Reinforcement Learning (RL) has become essential for eliciting complex reasoning capabilities in Large Language Models (LLMs). However, the substantial memory overhead of storing Key-Value (KV) caches during long-horizon rollouts acts as a critical bottleneck, often prohibiting efficient training on limited hardware. While existing KV compression techniques offer a remedy for inference, directly applying them to RL training induces a severe policy mismatch, leading to catastrophic performance collapse. To address this, we introduce Sparse-RL empowers stable RL training under sparse rollouts. We show that instability arises from a fundamental policy mismatch among the dense old policy, the sparse sampler policy, and the learner policy. To mitigate this issue, Sparse-RL incorporates Sparsity-Aware Rejection Sampling and Importance-based Reweighting to correct the off-policy bias introduced by compression-induced information loss. Experimental results show that Sparse-RL reduces rollout overhead compared to dense baselines while preserving the performance. Furthermore, Sparse-RL inherently implements sparsity-aware training, significantly enhancing model robustness during sparse inference deployment.
{
"annotation_id": "e7d0a106-eb35-48c8-98c6-692a85b9b34f",
"date_created": "2026-02-17T05:53:23.791000Z",
"date_modified": "2026-02-17T05:53:23.791000Z",
"file_hash": "7464d7213c6ad4928c140ad0e8dd6a2c69df90918c996c0f5b2dbcf750def847",
"private": false,
"record": {
"abstract": "Reinforcement Learning (RL) has become essential for eliciting complex reasoning capabilities in Large Language Models (LLMs). However, the substantial memory overhead of storing Key-Value (KV) caches during long-horizon rollouts acts as a critical bottleneck, often prohibiting efficient training on limited hardware. While existing KV compression techniques offer a remedy for inference, directly applying them to RL training induces a severe policy mismatch, leading to catastrophic performance collapse. To address this, we introduce Sparse-RL empowers stable RL training under sparse rollouts. We show that instability arises from a fundamental policy mismatch among the dense old policy, the sparse sampler policy, and the learner policy. To mitigate this issue, Sparse-RL incorporates Sparsity-Aware Rejection Sampling and Importance-based Reweighting to correct the off-policy bias introduced by compression-induced information loss. Experimental results show that Sparse-RL reduces rollout overhead compared to dense baselines while preserving the performance. Furthermore, Sparse-RL inherently implements sparsity-aware training, significantly enhancing model robustness during sparse inference deployment.",
"arxiv_id": "2601.10079",
"authors": [
"Sijia Luo",
"Xiaokang Zhang",
"Yuxuan Hu",
"Bohan Zhang",
"Ke Wang",
"Jinbo Su",
"Mengshu Sun",
"Lei Liang",
"Jing Zhang"
],
"categories": [
"cs.LG",
"cs.AI",
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts",
"url": "https://arxiv.org/abs/2601.10079",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "e6e75969-dfbf-420c-a322-a0145048437e",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}