dorsal/arxiv
View SchemaTALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees
| Authors | Tianyu Liu, Qitan Lv, Yuhao Shen, Xiao Sun, Xiaoyan Sun |
|---|---|
| Categories | |
| ArXiv ID | 2601.07353vv1 |
| URL | https://arxiv.org/abs/2601.07353 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Speculative decoding (SD) has become a standard technique for accelerating LLM inference without sacrificing output quality. Recent advances in speculative decoding have shifted from sequential chain-based drafting to tree-structured generation, where the draft model constructs a tree of candidate tokens to explore multiple possible drafts in parallel. However, existing tree-based SD methods typically build a fixed-width, fixed-depth draft tree, which fails to adapt to the varying difficulty of tokens and contexts. As a result, the draft model cannot dynamically adjust the tree structure to early stop on difficult tokens and extend generation for simple ones. To address these challenges, we introduce TALON, a training-free, budget-driven adaptive tree expansion framework that can be plugged into existing tree-based methods. Unlike static methods, TALON constructs the draft tree iteratively until a fixed token budget is met, using a hybrid expansion strategy that adaptively allocates the node budget to each layer of the draft tree. This framework naturally shapes the draft tree into a "deep-and-narrow" form for deterministic contexts and a "shallow-and-wide" form for uncertain branches, effectively optimizing the trade-off between exploration width and generation depth under a given budget. Extensive experiments across 5 models and 6 datasets demonstrate that TALON consistently outperforms state-of-the-art EAGLE-3, achieving up to 5.16x end-to-end speedup over auto-regressive decoding.
{
"annotation_id": "e6aa9808-d2fd-4f7a-b093-b3daa09feb06",
"date_created": "2026-02-17T05:53:11.817000Z",
"date_modified": "2026-02-17T05:53:11.817000Z",
"file_hash": "091030621cf92a07d9e5a4dcb8deb3946df3b6e9f9eb410d1c0f8d261ca3f832",
"private": false,
"record": {
"abstract": "Speculative decoding (SD) has become a standard technique for accelerating LLM inference without sacrificing output quality. Recent advances in speculative decoding have shifted from sequential chain-based drafting to tree-structured generation, where the draft model constructs a tree of candidate tokens to explore multiple possible drafts in parallel. However, existing tree-based SD methods typically build a fixed-width, fixed-depth draft tree, which fails to adapt to the varying difficulty of tokens and contexts. As a result, the draft model cannot dynamically adjust the tree structure to early stop on difficult tokens and extend generation for simple ones. To address these challenges, we introduce TALON, a training-free, budget-driven adaptive tree expansion framework that can be plugged into existing tree-based methods. Unlike static methods, TALON constructs the draft tree iteratively until a fixed token budget is met, using a hybrid expansion strategy that adaptively allocates the node budget to each layer of the draft tree. This framework naturally shapes the draft tree into a \"deep-and-narrow\" form for deterministic contexts and a \"shallow-and-wide\" form for uncertain branches, effectively optimizing the trade-off between exploration width and generation depth under a given budget. Extensive experiments across 5 models and 6 datasets demonstrate that TALON consistently outperforms state-of-the-art EAGLE-3, achieving up to 5.16x end-to-end speedup over auto-regressive decoding.",
"arxiv_id": "2601.07353",
"authors": [
"Tianyu Liu",
"Qitan Lv",
"Yuhao Shen",
"Xiao Sun",
"Xiaoyan Sun"
],
"categories": [
"cs.CL"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees",
"url": "https://arxiv.org/abs/2601.07353",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "b5c60170-35f0-43a5-bc5f-bdbe7a173ee1",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}