dorsal/arxiv
View SchemaModeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System
| Authors | Hsiang-Wei Huang, Junbin Lu, Kuang-Ming Chen, Jenq-Neng Hwang |
|---|---|
| Categories | |
| ArXiv ID | 2601.08829vv1 |
| URL | https://arxiv.org/abs/2601.08829 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
In this work, we explore the Large Language Model (LLM) agent reviewer dynamics in an Elo-ranked review system using real-world conference paper submissions. Multiple LLM agent reviewers with different personas are engage in multi round review interactions moderated by an Area Chair. We compare a baseline setting with conditions that incorporate Elo ratings and reviewer memory. Our simulation results showcase several interesting findings, including how incorporating Elo improves Area Chair decision accuracy, as well as reviewers' adaptive review strategy that exploits our Elo system without improving review effort. Our code is available at https://github.com/hsiangwei0903/EloReview.
{
"annotation_id": "1ac9ec7d-be14-498e-a8ff-46d336de3d7c",
"date_created": "2026-02-17T05:53:16.237000Z",
"date_modified": "2026-02-17T05:53:16.237000Z",
"file_hash": "ee46aacc4384c0fb80d3c321cfecda31452bd2c3aea1a3b4c79fc20edfe9f1a3",
"private": false,
"record": {
"abstract": "In this work, we explore the Large Language Model (LLM) agent reviewer dynamics in an Elo-ranked review system using real-world conference paper submissions. Multiple LLM agent reviewers with different personas are engage in multi round review interactions moderated by an Area Chair. We compare a baseline setting with conditions that incorporate Elo ratings and reviewer memory. Our simulation results showcase several interesting findings, including how incorporating Elo improves Area Chair decision accuracy, as well as reviewers\u0027 adaptive review strategy that exploits our Elo system without improving review effort. Our code is available at https://github.com/hsiangwei0903/EloReview.",
"arxiv_id": "2601.08829",
"authors": [
"Hsiang-Wei Huang",
"Junbin Lu",
"Kuang-Ming Chen",
"Jenq-Neng Hwang"
],
"categories": [
"cs.CL",
"cs.AI"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System",
"url": "https://arxiv.org/abs/2601.08829",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "06fed0c2-d212-44a3-9f22-c1dbafd530c8",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}