dorsal/arxiv
View SchemaFrom RAG to Agentic RAG for Faithful Islamic Question Answering
| Authors | Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz Al-Khatib, Logan Cochrane, Kareem Darwish, Rashid Yahiaoui, Firoj Alam |
|---|---|
| Categories | |
| ArXiv ID | 2601.07528vv1 |
| URL | https://arxiv.org/abs/2601.07528 |
| License | http://creativecommons.org/licenses/by-sa/4.0/ |
Abstract
LLMs are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations do not capture key real-world failure modes, notably free-form hallucinations and whether models appropriately abstain when evidence is lacking. To shed a light on this aspect we introduce ISLAMICFAITHQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modelling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur'an retrieval corpus of $\sim$6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We will make the experimental resources and datasets publicly available for the community.
{
"annotation_id": "91d3a256-a91b-45a5-8134-8411fc43dcf0",
"date_created": "2026-02-17T05:53:12.601000Z",
"date_modified": "2026-02-17T05:53:12.601000Z",
"file_hash": "7b174be250abc65bfe76345f3f0c07711998be91cc896ea2ccab7361d262456c",
"private": false,
"record": {
"abstract": "LLMs are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations do not capture key real-world failure modes, notably free-form hallucinations and whether models appropriately abstain when evidence is lacking. To shed a light on this aspect we introduce ISLAMICFAITHQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modelling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur\u0027an retrieval corpus of $\\sim$6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We will make the experimental resources and datasets publicly available for the community.",
"arxiv_id": "2601.07528",
"authors": [
"Gagan Bhatia",
"Hamdy Mubarak",
"Mustafa Jarrar",
"George Mikros",
"Fadi Zaraket",
"Mahmoud Alhirthani",
"Mutaz Al-Khatib",
"Logan Cochrane",
"Kareem Darwish",
"Rashid Yahiaoui",
"Firoj Alam"
],
"categories": [
"cs.CL",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by-sa/4.0/",
"title": "From RAG to Agentic RAG for Faithful Islamic Question Answering",
"url": "https://arxiv.org/abs/2601.07528",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "1ef024ed-b54f-4f37-a577-09cb7591178d",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}