dorsal/arxiv
View SchemaTreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG
| Authors | Tianhua Zhang, Kun Li, Junan Li, Yunxiang Li, Hongyin Luo, Xixin Wu, James Glass, Helen Meng |
|---|---|
| Categories | |
| ArXiv ID | 2601.06922vv1 |
| URL | https://arxiv.org/abs/2601.06922 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Agentic retrieval-augmented generation (RAG) formulates question answering as a multi-step interaction between reasoning and information retrieval, and has recently been advanced by reinforcement learning (RL) with outcome-based supervision. While effective, relying solely on sparse final rewards limits step-wise credit assignment and provides weak guidance for intermediate reasoning and actions. Recent efforts explore process-level supervision, but typically depend on offline constructed training data, which risks distribution shift, or require costly intermediate annotations. We present TreePS-RAG, an online, tree-based RL framework for agentic RAG that enables step-wise credit assignment while retaining standard outcome-only rewards. Our key insight is to model agentic RAG reasoning as a rollout tree, where each reasoning step naturally maps to a node. This tree structure allows step utility to be estimated via Monte Carlo estimation over its descendant outcomes, yielding fine-grained process advantages without requiring intermediate labels. To make this paradigm practical, we introduce an efficient online tree construction strategy that preserves exploration diversity under a constrained computational budget. With a rollout cost comparable to strong baselines like Search-R1, experiments on seven multi-hop and general QA benchmarks across multiple model scales show that TreePS-RAG consistently and significantly outperforms both outcome-supervised and leading process-supervised RL methods.
{
"annotation_id": "bb8da1ce-f176-44bd-a95d-898143cb0da5",
"date_created": "2026-02-17T05:53:07.895000Z",
"date_modified": "2026-02-17T05:53:07.895000Z",
"file_hash": "2f05e7d373ddf7a4eba7f3ef42ef9b1ad2134608069c1c71e402c783e1c5ae0f",
"private": false,
"record": {
"abstract": "Agentic retrieval-augmented generation (RAG) formulates question answering as a multi-step interaction between reasoning and information retrieval, and has recently been advanced by reinforcement learning (RL) with outcome-based supervision. While effective, relying solely on sparse final rewards limits step-wise credit assignment and provides weak guidance for intermediate reasoning and actions. Recent efforts explore process-level supervision, but typically depend on offline constructed training data, which risks distribution shift, or require costly intermediate annotations. We present TreePS-RAG, an online, tree-based RL framework for agentic RAG that enables step-wise credit assignment while retaining standard outcome-only rewards. Our key insight is to model agentic RAG reasoning as a rollout tree, where each reasoning step naturally maps to a node. This tree structure allows step utility to be estimated via Monte Carlo estimation over its descendant outcomes, yielding fine-grained process advantages without requiring intermediate labels. To make this paradigm practical, we introduce an efficient online tree construction strategy that preserves exploration diversity under a constrained computational budget. With a rollout cost comparable to strong baselines like Search-R1, experiments on seven multi-hop and general QA benchmarks across multiple model scales show that TreePS-RAG consistently and significantly outperforms both outcome-supervised and leading process-supervised RL methods.",
"arxiv_id": "2601.06922",
"authors": [
"Tianhua Zhang",
"Kun Li",
"Junan Li",
"Yunxiang Li",
"Hongyin Luo",
"Xixin Wu",
"James Glass",
"Helen Meng"
],
"categories": [
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG",
"url": "https://arxiv.org/abs/2601.06922",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "341213fe-0884-4d08-bade-c9bb15fd4c43",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}