dorsal/arxiv
View SchemaAPEX-SWE
| Authors | Abhi Kottamasu, Akul Datta, Aakash Barthwal, Chirag Mahapatra, Ajay Arun, Adarsh Hiremath, Brendan Foody, Bertie Vidgen |
|---|---|
| Categories | |
| ArXiv ID | 2601.08806vv1 |
| URL | https://arxiv.org/abs/2601.08806 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering work: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eight frontier models on APEX-SWE. Gemini 3 Pro (Thinking = High) performs best, with a Pass@1 score of 25\%. Our analysis shows that strong performance is primarily driven by epistemic reasoning, defined as the ability to distinguish between assumptions and verified facts, combined with agency to resolve uncertainty prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50).
{
"annotation_id": "eb5b86b5-2119-47ba-9065-7facd6248151",
"date_created": "2026-02-17T05:53:16.233000Z",
"date_modified": "2026-02-17T05:53:16.233000Z",
"file_hash": "d4fa45a9a0d32c15ddc2107a7983cd2230a45e34f25fe6a4692e34d6d5dbf87e",
"private": false,
"record": {
"abstract": "We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering work: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eight frontier models on APEX-SWE. Gemini 3 Pro (Thinking = High) performs best, with a Pass@1 score of 25\\%. Our analysis shows that strong performance is primarily driven by epistemic reasoning, defined as the ability to distinguish between assumptions and verified facts, combined with agency to resolve uncertainty prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50).",
"arxiv_id": "2601.08806",
"authors": [
"Abhi Kottamasu",
"Akul Datta",
"Aakash Barthwal",
"Chirag Mahapatra",
"Ajay Arun",
"Adarsh Hiremath",
"Brendan Foody",
"Bertie Vidgen"
],
"categories": [
"cs.SE",
"cs.AI",
"cs.CL"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "APEX-SWE",
"url": "https://arxiv.org/abs/2601.08806",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "2eda5001-aa13-4a0b-aab1-27ec86d8f776",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}