dorsal/arxiv
View SchemaSoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning
| Authors | Leo Fillioux, Omprakash Chakraborty, Ismail Ben Ayed, Paul-Henry Cournède, Stergios Christodoulidis, Maria Vakalopoulou, Jose Dolz |
|---|---|
| Categories | |
| ArXiv ID | 2601.08617vv1 |
| URL | https://arxiv.org/abs/2601.08617 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
With the increasing adoption of vision-language models (VLMs) in critical decision-making systems such as healthcare or autonomous driving, the calibration of their uncertainty estimates becomes paramount. Yet, this dimension has been largely underexplored in the VLM test-time prompt-tuning (TPT) literature, which has predominantly focused on improving their discriminative performance. Recent state-of-the-art advocates for enforcing full orthogonality over pairs of text prompt embeddings to enhance separability, and therefore calibration. Nevertheless, as we theoretically show in this work, the inherent gradients from fully orthogonal constraints will strongly push semantically related classes away, ultimately making the model overconfident. Based on our findings, we propose Semantic Orthogonal Calibration (SoC), a Huber-based regularizer that enforces smooth prototype separation while preserving semantic proximity, thereby improving calibration compared to prior orthogonality-based approaches. Across a comprehensive empirical validation, we demonstrate that SoC consistently improves calibration performance, while also maintaining competitive discriminative capabilities.
{
"annotation_id": "14f6582a-a83b-4940-ac4e-62177358bb18",
"date_created": "2026-02-17T05:53:15.904000Z",
"date_modified": "2026-02-17T05:53:15.904000Z",
"file_hash": "3b0ffe3bb15ca9c9d5e3ccacf971da9de95cff7759375d076382a94493fed252",
"private": false,
"record": {
"abstract": "With the increasing adoption of vision-language models (VLMs) in critical decision-making systems such as healthcare or autonomous driving, the calibration of their uncertainty estimates becomes paramount. Yet, this dimension has been largely underexplored in the VLM test-time prompt-tuning (TPT) literature, which has predominantly focused on improving their discriminative performance. Recent state-of-the-art advocates for enforcing full orthogonality over pairs of text prompt embeddings to enhance separability, and therefore calibration. Nevertheless, as we theoretically show in this work, the inherent gradients from fully orthogonal constraints will strongly push semantically related classes away, ultimately making the model overconfident. Based on our findings, we propose Semantic Orthogonal Calibration (SoC), a Huber-based regularizer that enforces smooth prototype separation while preserving semantic proximity, thereby improving calibration compared to prior orthogonality-based approaches. Across a comprehensive empirical validation, we demonstrate that SoC consistently improves calibration performance, while also maintaining competitive discriminative capabilities.",
"arxiv_id": "2601.08617",
"authors": [
"Leo Fillioux",
"Omprakash Chakraborty",
"Ismail Ben Ayed",
"Paul-Henry Courn\u00e8de",
"Stergios Christodoulidis",
"Maria Vakalopoulou",
"Jose Dolz"
],
"categories": [
"cs.CV"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning",
"url": "https://arxiv.org/abs/2601.08617",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "89a5c554-8d00-49e3-bb76-4cc24933d7c2",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}