dorsal/arxiv
View SchemaMM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval
| Authors | Abdelrahman Abdallah, Mohamed Darwish Mounis, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Mostafa Farouk Senussi, Mohamed Mahmoud, Mohammed Ali, Adam Jatowt, Hyun-Soo Kang |
|---|---|
| Categories | |
| ArXiv ID | 2601.09562vv2 |
| URL | https://arxiv.org/abs/2601.09562 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.
{
"annotation_id": "9672447e-ac9b-495f-a800-6c399641bc2b",
"date_created": "2026-02-17T05:53:19.952000Z",
"date_modified": "2026-02-17T05:53:19.952000Z",
"file_hash": "a7ed29edeedc99ab8e99cc22c6faf33db87e7a6ed13b1385c35d799baf7a2419",
"private": false,
"record": {
"abstract": "Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.",
"arxiv_id": "2601.09562",
"authors": [
"Abdelrahman Abdallah",
"Mohamed Darwish Mounis",
"Mahmoud Abdalla",
"Mahmoud SalahEldin Kasem",
"Mostafa Farouk Senussi",
"Mohamed Mahmoud",
"Mohammed Ali",
"Adam Jatowt",
"Hyun-Soo Kang"
],
"categories": [
"cs.IR"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval",
"url": "https://arxiv.org/abs/2601.09562",
"version": "v2"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "5db329e4-dee1-446c-a223-64bb24f5d73c",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}