dorsal/arxiv
View SchemaFaTRQ: Tiered Residual Quantization for LLM Vector Search in Far-Memory-Aware ANNS Systems
| Authors | Tianqi Zhang, Flavio Ponzina, Tajana Rosing |
|---|---|
| Categories | |
| ArXiv ID | 2601.09985vv1 |
| URL | https://arxiv.org/abs/2601.09985 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Approximate Nearest-Neighbor Search (ANNS) is a key technique in retrieval-augmented generation (RAG), enabling rapid identification of the most relevant high-dimensional embeddings from massive vector databases. Modern ANNS engines accelerate this process using prebuilt indexes and store compressed vector-quantized representations in fast memory. However, they still rely on a costly second-pass refinement stage that reads full-precision vectors from slower storage like SSDs. For modern text and multimodal embeddings, these reads now dominate the latency of the entire query. We propose FaTRQ, a far-memory-aware refinement system using tiered memory that eliminates the need to fetch full vectors from storage. It introduces a progressive distance estimator that refines coarse scores using compact residuals streamed from far memory. Refinement stops early once a candidate is provably outside the top-k. To support this, we propose tiered residual quantization, which encodes residuals as ternary values stored efficiently in far memory. A custom accelerator is deployed in a CXL Type-2 device to perform low-latency refinement locally. Together, FaTRQ improves the storage efficiency by 2.4$\times$ and improves the throughput by up to 9$ \times$ than SOTA GPU ANNS system.
{
"annotation_id": "498c2f2b-f6c6-4320-9803-4c76153c0c98",
"date_created": "2026-02-17T05:53:24.223000Z",
"date_modified": "2026-02-17T05:53:24.223000Z",
"file_hash": "1ee5ca777563311cfae42cdef1a5aee182d0060b31d8ab220ef9a464006f927a",
"private": false,
"record": {
"abstract": "Approximate Nearest-Neighbor Search (ANNS) is a key technique in retrieval-augmented generation (RAG), enabling rapid identification of the most relevant high-dimensional embeddings from massive vector databases. Modern ANNS engines accelerate this process using prebuilt indexes and store compressed vector-quantized representations in fast memory. However, they still rely on a costly second-pass refinement stage that reads full-precision vectors from slower storage like SSDs. For modern text and multimodal embeddings, these reads now dominate the latency of the entire query. We propose FaTRQ, a far-memory-aware refinement system using tiered memory that eliminates the need to fetch full vectors from storage. It introduces a progressive distance estimator that refines coarse scores using compact residuals streamed from far memory. Refinement stops early once a candidate is provably outside the top-k. To support this, we propose tiered residual quantization, which encodes residuals as ternary values stored efficiently in far memory. A custom accelerator is deployed in a CXL Type-2 device to perform low-latency refinement locally. Together, FaTRQ improves the storage efficiency by 2.4$\\times$ and improves the throughput by up to 9$ \\times$ than SOTA GPU ANNS system.",
"arxiv_id": "2601.09985",
"authors": [
"Tianqi Zhang",
"Flavio Ponzina",
"Tajana Rosing"
],
"categories": [
"cs.LG",
"cs.AR",
"cs.IR"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "FaTRQ: Tiered Residual Quantization for LLM Vector Search in Far-Memory-Aware ANNS Systems",
"url": "https://arxiv.org/abs/2601.09985",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "5c563293-a58e-4b5a-83cf-b21e7def736d",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}