dorsal/arxiv
View SchemaThe Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments
| Authors | Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, Edwin Chen |
|---|---|
| Categories | |
| ArXiv ID | 2601.09032vv1 |
| URL | https://arxiv.org/abs/2601.09032 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
The advancement of large language model (LLM) based agents has shifted AI evaluation from single-turn response assessment to multi-step task completion in interactive environments. We present an empirical study evaluating frontier AI models on 150 workplace tasks within a realistic e-commerce RL environment from Surge. Our analysis reveals an empirically-derived \emph{hierarchy of agentic capabilities} that models must master for real-world deployment: (1) tool use, (2) planning and goal formation, (3) adaptability, (4) groundedness, and (5) common-sense reasoning. Even the best-performing models fail approximately 40\% of the tasks, with failures clustering predictably along this hierarchy. Weaker models struggle with fundamental tool use and planning, whereas stronger models primarily fail on tasks requiring contextual inference beyond explicit instructions. We introduce a task-centric design methodology for RL environments that emphasizes diversity and domain expert contributions, provide detailed failure analysis, and discuss implications for agent development. Our findings suggest that while current frontier models can demonstrate coherent multi-step behavior, substantial capability gaps remain before achieving human-level task completion in realistic workplace settings.
{
"annotation_id": "d01d52d2-e981-4a88-9355-c6e044c37ccf",
"date_created": "2026-02-17T05:53:20.473000Z",
"date_modified": "2026-02-17T05:53:20.473000Z",
"file_hash": "3f8056e0bb613061f93002845c178f513f54314978d4c289fa9b504e1e7732a5",
"private": false,
"record": {
"abstract": "The advancement of large language model (LLM) based agents has shifted AI evaluation from single-turn response assessment to multi-step task completion in interactive environments. We present an empirical study evaluating frontier AI models on 150 workplace tasks within a realistic e-commerce RL environment from Surge. Our analysis reveals an empirically-derived \\emph{hierarchy of agentic capabilities} that models must master for real-world deployment: (1) tool use, (2) planning and goal formation, (3) adaptability, (4) groundedness, and (5) common-sense reasoning. Even the best-performing models fail approximately 40\\% of the tasks, with failures clustering predictably along this hierarchy. Weaker models struggle with fundamental tool use and planning, whereas stronger models primarily fail on tasks requiring contextual inference beyond explicit instructions. We introduce a task-centric design methodology for RL environments that emphasizes diversity and domain expert contributions, provide detailed failure analysis, and discuss implications for agent development. Our findings suggest that while current frontier models can demonstrate coherent multi-step behavior, substantial capability gaps remain before achieving human-level task completion in realistic workplace settings.",
"arxiv_id": "2601.09032",
"authors": [
"Logan Ritchie",
"Sushant Mehta",
"Nick Heiner",
"Mason Yu",
"Edwin Chen"
],
"categories": [
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments",
"url": "https://arxiv.org/abs/2601.09032",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "9adb8220-c232-4fef-b6e1-b39d541a6108",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}