dorsal/arxiv
View SchemaEmotional Support Evaluation Framework via Controllable and Diverse Seeker Simulator
| Authors | Chaewon Heo, Cheyon Jin, Yohan Jo |
|---|---|
| Categories | |
| ArXiv ID | 2601.07698vv1 |
| URL | https://arxiv.org/abs/2601.07698 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged: use help-seeker simulators to interact with supporter chatbots. However, current simulators suffer from two critical limitations: (1) they fail to capture the behavioral diversity of real-world seekers, often portraying them as overly cooperative, and (2) they lack the controllability required to simulate specific seeker profiles. To address these challenges, we present a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior. Using authentic Reddit conversations, we train our model via a Mixture-of-Experts (MoE) architecture, which effectively differentiates diverse seeker behaviors into specialized parameter subspaces, thereby enhancing fine-grained controllability. Our simulator achieves superior profile adherence and behavioral diversity compared to existing approaches. Furthermore, evaluating 7 prominent supporter models with our system uncovers previously obscured performance degradations. These findings underscore the utility of our framework in providing a more faithful and stress-tested evaluation for emotional support chatbots.
{
"annotation_id": "302a4dc5-b8de-4bd8-bf04-1b8c75b71bc9",
"date_created": "2026-02-17T05:53:12.265000Z",
"date_modified": "2026-02-17T05:53:12.265000Z",
"file_hash": "d5c4458b0b277409b0dbcef5d46b567e20f36ebdf3790ad44a49a1615b6fe64b",
"private": false,
"record": {
"abstract": "As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged: use help-seeker simulators to interact with supporter chatbots. However, current simulators suffer from two critical limitations: (1) they fail to capture the behavioral diversity of real-world seekers, often portraying them as overly cooperative, and (2) they lack the controllability required to simulate specific seeker profiles. To address these challenges, we present a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior. Using authentic Reddit conversations, we train our model via a Mixture-of-Experts (MoE) architecture, which effectively differentiates diverse seeker behaviors into specialized parameter subspaces, thereby enhancing fine-grained controllability. Our simulator achieves superior profile adherence and behavioral diversity compared to existing approaches. Furthermore, evaluating 7 prominent supporter models with our system uncovers previously obscured performance degradations. These findings underscore the utility of our framework in providing a more faithful and stress-tested evaluation for emotional support chatbots.",
"arxiv_id": "2601.07698",
"authors": [
"Chaewon Heo",
"Cheyon Jin",
"Yohan Jo"
],
"categories": [
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "Emotional Support Evaluation Framework via Controllable and Diverse Seeker Simulator",
"url": "https://arxiv.org/abs/2601.07698",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "725e27fe-3ab6-424f-8792-60ff6b6dd3c1",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}