dorsal/arxiv
View SchemaHow Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction
| Authors | Yingjie He, Zhaolu Kang, Kehan Jiang, Qianyuan Zhang, Jiachen Qian, Chunlei Meng, Yujie Feng, Yuan Wang, Jiabao Dou, Aming Wu, Leqi Zheng, Pengxiang Zhao, Jiaxin Liu, Zeyu Zhang, Lei Wang, Guansu Wang, Qishi Zhan, Xiaomin He, Meisheng Zhang, Jianyuan Ni |
|---|---|
| Categories | |
| ArXiv ID | 2601.08626vv1 |
| URL | https://arxiv.org/abs/2601.08626 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is ill-posed for automated evaluation because multiple valid word orders often exist. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, consistency, robustness sensitivity, and information density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent dissociation between semantic recall and structural planning, suggesting that structural robustness is not an automatic byproduct of semantic competence.
{
"annotation_id": "8a2ebd72-b39b-43fd-8aec-c887754a0ea3",
"date_created": "2026-02-17T05:53:16.143000Z",
"date_modified": "2026-02-17T05:53:16.143000Z",
"file_hash": "e989f24b2b3bba9198013aa9b3e8c3cd7dc3a99c4abc38e99b9addd796efe0e5",
"private": false,
"record": {
"abstract": "Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is ill-posed for automated evaluation because multiple valid word orders often exist. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, consistency, robustness sensitivity, and information density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent dissociation between semantic recall and structural planning, suggesting that structural robustness is not an automatic byproduct of semantic competence.",
"arxiv_id": "2601.08626",
"authors": [
"Yingjie He",
"Zhaolu Kang",
"Kehan Jiang",
"Qianyuan Zhang",
"Jiachen Qian",
"Chunlei Meng",
"Yujie Feng",
"Yuan Wang",
"Jiabao Dou",
"Aming Wu",
"Leqi Zheng",
"Pengxiang Zhao",
"Jiaxin Liu",
"Zeyu Zhang",
"Lei Wang",
"Guansu Wang",
"Qishi Zhan",
"Xiaomin He",
"Meisheng Zhang",
"Jianyuan Ni"
],
"categories": [
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction",
"url": "https://arxiv.org/abs/2601.08626",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "f14f88b1-def0-4075-b90f-efe8ca3b69c5",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}