dorsal/arxiv
View SchemaApplying MDL to Learning Best Model Granularity
| Authors | Qiong Gao, Ming Li, Paul Vitanyi |
|---|---|
| Categories | |
| ArXiv ID | physics/0005062 |
| URL | https://arxiv.org/abs/physics/0005062 |
Abstract
The Minimum Description Length (MDL) principle is solidly based on a provably ideal method of inference using Kolmogorov complexity. We test how the theory behaves in practice on a general problem in model selection: that of learning the best model granularity. The performance of a model depends critically on the granularity, for example the choice of precision of the parameters. Too high precision generally involves modeling of accidental noise and too low precision may lead to confusion of models that should be distinguished. This precision is often determined ad hoc. In MDL the best model is the one that most compresses a two-part code of the data set: this embodies ``Occam's Razor.'' In two quite different experimental settings the theoretical value determined using MDL coincides with the best value found experimentally. In the first experiment the task is to recognize isolated handwritten characters in one subject's handwriting, irrespective of size and orientation. Based on a new modification of elastic matching, using multiple prototypes per character, the optimal prediction rate is predicted for the learned parameter (length of sampling interval) considered most likely by MDL, which is shown to coincide with the best value found experimentally. In the second experiment the task is to model a robot arm with two degrees of freedom using a three layer feed-forward neural network where we need to determine the number of nodes in the hidden layer giving best modeling performance. The optimal model (the one that extrapolizes best on unseen examples) is predicted for the number of nodes in the hidden layer considered most likely by MDL, which again is found to coincide with the best value found experimentally.
{
"annotation_id": "fb2552bb-d5aa-4dd9-86d1-03b317d46233",
"date_created": "2026-03-02T18:00:32.290000Z",
"date_modified": "2026-03-02T18:00:32.290000Z",
"file_hash": "8628bb6c6135a41a09d25981bda2684a98b5f7f386a6d3d586516cf43b436d54",
"private": false,
"record": {
"abstract": "The Minimum Description Length (MDL) principle is solidly based on a provably\nideal method of inference using Kolmogorov complexity. We test how the theory\nbehaves in practice on a general problem in model selection: that of learning\nthe best model granularity. The performance of a model depends critically on\nthe granularity, for example the choice of precision of the parameters. Too\nhigh precision generally involves modeling of accidental noise and too low\nprecision may lead to confusion of models that should be distinguished. This\nprecision is often determined ad hoc. In MDL the best model is the one that\nmost compresses a two-part code of the data set: this embodies ``Occam\u0027s\nRazor.\u0027\u0027 In two quite different experimental settings the theoretical value\ndetermined using MDL coincides with the best value found experimentally. In the\nfirst experiment the task is to recognize isolated handwritten characters in\none subject\u0027s handwriting, irrespective of size and orientation. Based on a new\nmodification of elastic matching, using multiple prototypes per character, the\noptimal prediction rate is predicted for the learned parameter (length of\nsampling interval) considered most likely by MDL, which is shown to coincide\nwith the best value found experimentally. In the second experiment the task is\nto model a robot arm with two degrees of freedom using a three layer\nfeed-forward neural network where we need to determine the number of nodes in\nthe hidden layer giving best modeling performance. The optimal model (the one\nthat extrapolizes best on unseen examples) is predicted for the number of nodes\nin the hidden layer considered most likely by MDL, which again is found to\ncoincide with the best value found experimentally.",
"arxiv_id": "physics/0005062",
"authors": [
"Qiong Gao",
"Ming Li",
"Paul Vitanyi"
],
"categories": [
"physics.data-an",
"cs.AI",
"cs.CV"
],
"title": "Applying MDL to Learning Best Model Granularity",
"url": "https://arxiv.org/abs/physics/0005062"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "b2f22a0a-379c-4342-8623-fd1209351714",
"id": "arXiv Dataset IDs",
"type": "Model",
"variant": "snapshot-2026-03-01",
"version": "0.1.0"
},
"user_id": 1000002
}