dorsal/arxiv
View SchemaBeyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learning
| Authors | Yanan Chen, Tieliang Gong, Yunjiao Zhang, Wen Wen |
|---|---|
| Categories | |
| ArXiv ID | 2601.07636vv1 |
| URL | https://arxiv.org/abs/2601.07636 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Continual Learning (CL) aims to enable models to sequentially learn multiple tasks without forgetting previous knowledge. Recent studies have shown that optimizing towards flatter loss minima can improve model generalization. However, existing sharpness-aware methods for CL suffer from two key limitations: (1) they treat sharpness regularization as a unified signal without distinguishing the contributions of its components. and (2) they introduce substantial computational overhead that impedes practical deployment. To address these challenges, we propose FLAD, a novel optimization framework that decomposes sharpness-aware perturbations into gradient-aligned and stochastic-noise components, and show that retaining only the noise component promotes generalization. We further introduce a lightweight scheduling scheme that enables FLAD to maintain significant performance gains even under constrained training time. FLAD can be seamlessly integrated into various CL paradigms and consistently outperforms standard and sharpness-aware optimizers in diverse experimental settings, demonstrating its effectiveness and practicality in CL.
{
"annotation_id": "3cc828c5-0720-4eb7-a675-c036543d160a",
"date_created": "2026-02-17T05:53:11.625000Z",
"date_modified": "2026-02-17T05:53:11.625000Z",
"file_hash": "312ec10db1e68e24cde661592a53f0b0888e9419d1d48853a9b9d73c53f660f7",
"private": false,
"record": {
"abstract": "Continual Learning (CL) aims to enable models to sequentially learn multiple tasks without forgetting previous knowledge. Recent studies have shown that optimizing towards flatter loss minima can improve model generalization. However, existing sharpness-aware methods for CL suffer from two key limitations: (1) they treat sharpness regularization as a unified signal without distinguishing the contributions of its components. and (2) they introduce substantial computational overhead that impedes practical deployment. To address these challenges, we propose FLAD, a novel optimization framework that decomposes sharpness-aware perturbations into gradient-aligned and stochastic-noise components, and show that retaining only the noise component promotes generalization. We further introduce a lightweight scheduling scheme that enables FLAD to maintain significant performance gains even under constrained training time. FLAD can be seamlessly integrated into various CL paradigms and consistently outperforms standard and sharpness-aware optimizers in diverse experimental settings, demonstrating its effectiveness and practicality in CL.",
"arxiv_id": "2601.07636",
"authors": [
"Yanan Chen",
"Tieliang Gong",
"Yunjiao Zhang",
"Wen Wen"
],
"categories": [
"cs.LG"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "Beyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learning",
"url": "https://arxiv.org/abs/2601.07636",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "58c6b944-b29d-4b9e-8ce3-5ab916f5483b",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}