dorsal/arxiv
View SchemaFine Grained Evaluation of LLMs-as-Judges
| Authors | Sourav Saha, Mandar Mitra |
|---|---|
| Categories | |
| ArXiv ID | 2601.08919vv1 |
| URL | https://arxiv.org/abs/2601.08919 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
A good deal of recent research has focused on how Large Language Models (LLMs) may be used as `judges' in place of humans to evaluate the quality of the output produced by various text / image processing systems. Within this broader context, a number of studies have investigated the specific question of how effectively LLMs can be used as relevance assessors for the standard ad hoc task in Information Retrieval (IR). We extend these studies by looking at additional questions. Most importantly, we use a Wikipedia based test collection created by the INEX initiative, and prompt LLMs to not only judge whether documents are relevant / non-relevant, but to highlight relevant passages in documents that it regards as useful. The human relevance assessors involved in creating this collection were given analogous instructions, i.e., they were asked to highlight all passages within a document that respond to the information need expressed in a query. This enables us to evaluate the quality of LLMs as judges not only at the document level, but to also quantify how often these `judges' are right for the right reasons. Our findings suggest that LLMs-as-judges work best under human supervision.
{
"annotation_id": "9d15e88e-6521-4238-9966-920ea2a97f2c",
"date_created": "2026-02-17T05:53:19.285000Z",
"date_modified": "2026-02-17T05:53:19.285000Z",
"file_hash": "7358a401a22248bc52299b209b80a20e24a08bab365f84c42d08b95f1d9e1f70",
"private": false,
"record": {
"abstract": "A good deal of recent research has focused on how Large Language Models\n (LLMs) may be used as `judges\u0027 in place of humans to evaluate the quality\n of the output produced by various text / image processing systems. Within\n this broader context, a number of studies have investigated the specific\n question of how effectively LLMs can be used as relevance assessors for\n the standard ad hoc task in Information Retrieval (IR). We extend these\n studies by looking at additional questions. Most importantly, we use a\n Wikipedia based test collection created by the INEX initiative, and\n prompt LLMs to not only judge whether documents are relevant /\n non-relevant, but to highlight relevant passages in documents that it\n regards as useful. The human relevance assessors involved in creating\n this collection were given analogous instructions, i.e., they were asked\n to highlight all passages within a document that respond to the\n information need expressed in a query. This enables us to evaluate the\n quality of LLMs as judges not only at the document level, but to also\n quantify how often these `judges\u0027 are right for the right reasons.\n Our findings suggest that LLMs-as-judges work best under human\n supervision.",
"arxiv_id": "2601.08919",
"authors": [
"Sourav Saha",
"Mandar Mitra"
],
"categories": [
"cs.IR",
"cs.CL",
"cs.LG"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Fine Grained Evaluation of LLMs-as-Judges",
"url": "https://arxiv.org/abs/2601.08919",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "c185ee52-1c2d-47f1-bf4a-47fe4fbb89d6",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}