dorsal/arxiv
View SchemaHierarchical Sparse Plus Low Rank Compression of LLM
| Authors | Pawan Kumar, Aditi Gupta |
|---|---|
| Categories | |
| ArXiv ID | 2601.07839vv1 |
| URL | https://arxiv.org/abs/2601.07839 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Modern large language models (LLMs) place extraordinary pressure on memory and compute budgets, making principled compression indispensable for both deployment and continued training. We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that (i) removes the largest-magnitude weights into a sparse matrix S and (ii) applies a recursive Hierarchically Sparse Separable (HSS) low-rank factorisation to the dense residual matrix. A recursive rank-reducing strategy and a reverse Cuthill-Mckee (RCM) permutation are introduced to align high weights towards the diagonal with the block-diagonal hierarchy, maximising off-diagonal compressibility (because they are touched only once). HSS is hardware-friendly: its matrix-vector multiply reduces to one sparse and a sequence of thin-matrix multiplications and can be trained end-to-end with standard optimisers. Experiments on LLaMA-7B show that targeting only the self-attention projections (1.6 B parameters of Q, K, and V matrices out of a total 7B parameters) suffices to yield large memory savings while retaining comparable state-of-the-art perplexity scores on test samples of the WikiText dataset. For example, with a 30\% sparsity budget and an outer rank of 512, sHSS-RCM achieves a perplexity of 1.64, outperforming dense baselines and classical sparse-plus-SVD variants, while also achieving significant memory savings.
{
"annotation_id": "3574b367-6b91-4fd2-92a6-66c88259adc6",
"date_created": "2026-02-17T05:53:12.444000Z",
"date_modified": "2026-02-17T05:53:12.444000Z",
"file_hash": "1daabd511260ab5ff860be9cb9cebaea1b2f047ca8ac0c96cb56a74f2b584b41",
"private": false,
"record": {
"abstract": "Modern large language models (LLMs) place extraordinary pressure on memory and compute budgets, making principled compression indispensable for both deployment and continued training. We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that (i) removes the largest-magnitude weights into a sparse matrix S and (ii) applies a recursive Hierarchically Sparse Separable (HSS) low-rank factorisation to the dense residual matrix. A recursive rank-reducing strategy and a reverse Cuthill-Mckee (RCM) permutation are introduced to align high weights towards the diagonal with the block-diagonal hierarchy, maximising off-diagonal compressibility (because they are touched only once). HSS is hardware-friendly: its matrix-vector multiply reduces to one sparse and a sequence of thin-matrix multiplications and can be trained end-to-end with standard optimisers.\n Experiments on LLaMA-7B show that targeting only the self-attention projections (1.6 B parameters of Q, K, and V matrices out of a total 7B parameters) suffices to yield large memory savings while retaining comparable state-of-the-art perplexity scores on test samples of the WikiText dataset. For example, with a 30\\% sparsity budget and an outer rank of 512, sHSS-RCM achieves a perplexity of 1.64, outperforming dense baselines and classical sparse-plus-SVD variants, while also achieving significant memory savings.",
"arxiv_id": "2601.07839",
"authors": [
"Pawan Kumar",
"Aditi Gupta"
],
"categories": [
"cs.LG",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Hierarchical Sparse Plus Low Rank Compression of LLM",
"url": "https://arxiv.org/abs/2601.07839",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "15f4eaf6-3490-4890-ac2f-c6fe7aad7e53",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}