dorsal/arxiv
View SchemaMinimum Entropy Aproach to Word Segmentation Problems
| Authors | Bin Wang |
|---|---|
| Categories | |
| ArXiv ID | physics/0008232 |
| URL | https://arxiv.org/abs/physics/0008232 |
| DOI | 10.1016/S0378-4371(00)00545-8 |
Abstract
Given a sequence composed of a limit number of characters, we try to "read" it as a "text". This involves to segment the sequence into "words". The difficulty is to distinguish good segmentation from enormous number of random ones.Aiming at revealing the nonrandomness of the sequence as strongly as possible, by applying maximum likelihood method, we find a quantity called Segmentation Entropy that can be used to fulfill the duty. Contrary to commonplace where maximum entropy principle was applied to obtain good solution, we choose to {\em minimize} the segmentation entropy to obtain good segmentation. The concept developed in this letter can be used to study the noncoding DNA sequences, e.g., for regulatory elements prediction, in eukaryote genomes.
{
"annotation_id": "359e83fe-4cf9-449a-a178-bb9ec7c13363",
"date_created": "2026-03-02T18:00:32.420000Z",
"date_modified": "2026-03-02T18:00:32.420000Z",
"file_hash": "83d82e9563ec3f2e75517b2dcee84cc7c0c29254467f144d9d1f72d0c0df8fcf",
"private": false,
"record": {
"abstract": "Given a sequence composed of a limit number of characters, we try to \"read\"\nit as a \"text\". This involves to segment the sequence into \"words\". The\ndifficulty is to distinguish good segmentation from enormous number of random\nones.Aiming at revealing the nonrandomness of the sequence as strongly as\npossible, by applying maximum likelihood method, we find a quantity called\nSegmentation Entropy that can be used to fulfill the duty. Contrary to\ncommonplace where maximum entropy principle was applied to obtain good\nsolution, we choose to {\\em minimize} the segmentation entropy to obtain good\nsegmentation. The concept developed in this letter can be used to study the\nnoncoding DNA sequences, e.g., for regulatory elements prediction, in eukaryote\ngenomes.",
"arxiv_id": "physics/0008232",
"authors": [
"Bin Wang"
],
"categories": [
"physics.bio-ph",
"cond-mat.stat-mech",
"physics.data-an",
"q-bio"
],
"doi": "10.1016/S0378-4371(00)00545-8",
"title": "Minimum Entropy Aproach to Word Segmentation Problems",
"url": "https://arxiv.org/abs/physics/0008232"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "8241ce94-93ae-4ac5-929c-5debc6f3439a",
"id": "arXiv Dataset IDs",
"type": "Model",
"variant": "snapshot-2026-03-01",
"version": "0.1.0"
},
"user_id": 1000002
}