dorsal/arxiv
View SchemaEntropy and inference, revisited
| Authors | Ilya Nemenman, Fariel Shafee, William Bialek |
|---|---|
| Categories | |
| ArXiv ID | physics/0108025 |
| URL | https://arxiv.org/abs/physics/0108025 |
| Journal | Advances in Neural Information Processing Systems 14, 2002. MIT Press |
Abstract
We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.
{
"annotation_id": "e376410a-cebe-4da8-9596-a18321260d1d",
"date_created": "2026-03-02T18:00:36.340000Z",
"date_modified": "2026-03-02T18:00:36.340000Z",
"file_hash": "604d3dbefda7e5bceff4cd2031775547f084333af5aaf6b49de8be179916acda",
"private": false,
"record": {
"abstract": "We study properties of popular near-uniform (Dirichlet) priors for learning\nundersampled probability distributions on discrete nonmetric spaces and show\nthat they lead to disastrous results. However, an Occam-style phase space\nargument expands the priors into their infinite mixture and resolves most of\nthe observed problems. This leads to a surprisingly good estimator of entropies\nof discrete distributions.",
"arxiv_id": "physics/0108025",
"authors": [
"Ilya Nemenman",
"Fariel Shafee",
"William Bialek"
],
"categories": [
"physics.data-an"
],
"journal_ref": "Advances in Neural Information Processing Systems 14, 2002. MIT\n Press",
"title": "Entropy and inference, revisited",
"url": "https://arxiv.org/abs/physics/0108025"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "4ebf74f8-be20-4e13-a4d8-8dffb1627d2f",
"id": "arXiv Dataset IDs",
"type": "Model",
"variant": "snapshot-2026-03-01",
"version": "0.1.0"
},
"user_id": 1000002
}