dorsal/arxiv
View SchemaMM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval
| Authors | Abdelrahman Abdallah, Mohamed Darwish Mounis, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Mostafa Farouk Senussi, Mohamed Mahmoud, Mohammed Ali, Adam Jatowt, Hyun-Soo Kang |
|---|---|
| Categories | |
| ArXiv ID | 2601.09562vv1 |
| URL | https://arxiv.org/abs/2601.09562 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.
{
"annotation_id": "e018adc0-b474-4e35-a0c1-69b198713586",
"date_created": "2026-02-17T05:53:19.951000Z",
"date_modified": "2026-02-17T05:53:19.951000Z",
"file_hash": "74951c8c4cdbc1c82a035634f4bf13831e1d10d0e74e313a70a416ba9453edc1",
"private": false,
"record": {
"abstract": "Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.",
"arxiv_id": "2601.09562",
"authors": [
"Abdelrahman Abdallah",
"Mohamed Darwish Mounis",
"Mahmoud Abdalla",
"Mahmoud SalahEldin Kasem",
"Mostafa Farouk Senussi",
"Mohamed Mahmoud",
"Mohammed Ali",
"Adam Jatowt",
"Hyun-Soo Kang"
],
"categories": [
"cs.IR"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval",
"url": "https://arxiv.org/abs/2601.09562",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "1529ea74-fb44-4a1e-a987-ab79a2fcec47",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}