dorsal/arxiv
View SchemaSurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability
| Authors | Ruochen Li, Kun Yuan, Yufei Xia, Yue Zhou, Qingyu Lu, Weihang Li, Youxiang Zhu, Nassir Navab |
|---|---|
| Categories | |
| ArXiv ID | 2601.10455vv1 |
| URL | https://arxiv.org/abs/2601.10455 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Surgical planning integrates visual perception, long-horizon reasoning, and procedural knowledge, yet it remains unclear whether current evaluation protocols reliably assess vision-language models (VLMs) in safety-critical settings. Motivated by a goal-oriented view of surgical planning, we define planning correctness via phase-goal satisfiability, where plan validity is determined by expert-defined surgical rules. Based on this definition, we introduce a multicentric meta-evaluation benchmark with valid procedural variations and invalid plans containing order and content errors. Using this benchmark, we show that sequence similarity metrics systematically misjudge planning quality, penalizing valid plans while failing to identify invalid ones. We therefore adopt a rule-based goal-satisfiability metric as a high-precision meta-evaluation reference to assess Video-LLMs under progressively constrained settings, revealing failures due to perception errors and under-constrained reasoning. Structural knowledge consistently improves performance, whereas semantic guidance alone is unreliable and benefits larger models only when combined with structural constraints.
{
"annotation_id": "c5b329b0-d187-458f-ae46-d9fd3e8998cd",
"date_created": "2026-02-17T05:53:23.414000Z",
"date_modified": "2026-02-17T05:53:23.414000Z",
"file_hash": "6a4795f85870e010f327d1316ccdbec053f4161758092fe836c804871bfc413c",
"private": false,
"record": {
"abstract": "Surgical planning integrates visual perception, long-horizon reasoning, and procedural knowledge, yet it remains unclear whether current evaluation protocols reliably assess vision-language models (VLMs) in safety-critical settings. Motivated by a goal-oriented view of surgical planning, we define planning correctness via phase-goal satisfiability, where plan validity is determined by expert-defined surgical rules. Based on this definition, we introduce a multicentric meta-evaluation benchmark with valid procedural variations and invalid plans containing order and content errors. Using this benchmark, we show that sequence similarity metrics systematically misjudge planning quality, penalizing valid plans while failing to identify invalid ones. We therefore adopt a rule-based goal-satisfiability metric as a high-precision meta-evaluation reference to assess Video-LLMs under progressively constrained settings, revealing failures due to perception errors and under-constrained reasoning. Structural knowledge consistently improves performance, whereas semantic guidance alone is unreliable and benefits larger models only when combined with structural constraints.",
"arxiv_id": "2601.10455",
"authors": [
"Ruochen Li",
"Kun Yuan",
"Yufei Xia",
"Yue Zhou",
"Qingyu Lu",
"Weihang Li",
"Youxiang Zhu",
"Nassir Navab"
],
"categories": [
"cs.CL",
"cs.RO"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability",
"url": "https://arxiv.org/abs/2601.10455",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "fbdab420-f140-486a-a9c0-8afd2fd7eb0a",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}