dorsal/arxiv
View SchemaPublication Bias (The "File-Drawer Problem") in Scientific Inference
| Authors | Jeffrey D. Scargle |
|---|---|
| Categories | |
| ArXiv ID | physics/9909033 |
| URL | https://arxiv.org/abs/physics/9909033 |
Abstract
Publication bias arises whenever the probability that a study is published depends on the statistical significance of its results. This bias, often called the file-drawer effect since the unpublished results are imagined to be tucked away in researchers' file cabinets, is potentially a severe impediment to combining the statistical results of studies collected from the literature. With almost any reasonable quantitative model for publication bias, only a small number of studies lost in the file-drawer will produce a significant bias. This result contradicts the well known Fail Safe File Drawer (FSFD) method for setting limits on the potential harm of publication bias, widely used in social, medical and psychic research. This method incorrectly treats the file drawer as unbiased, and almost always misestimates the seriousness of publication bias. A large body of not only psychic research, but medical and social science studies, has mistakenly relied on this method to validate claimed discoveries. Statistical combination can be trusted only if it is known with certainty that all studies that have been carried out are included. Such certainty is virtually impossible to achieve in literature surveys.
{
"annotation_id": "dc4b4fa7-43b3-434d-9d46-b6f7dd54a6b8",
"date_created": "2026-03-02T18:01:25.178000Z",
"date_modified": "2026-03-02T18:01:25.178000Z",
"file_hash": "9238bd9b09715235120bc02147d25234ee390de9cd2ce161ec8bbcb4ec076f43",
"private": false,
"record": {
"abstract": "Publication bias arises whenever the probability that a study is published\ndepends on the statistical significance of its results. This bias, often called\nthe file-drawer effect since the unpublished results are imagined to be tucked\naway in researchers\u0027 file cabinets, is potentially a severe impediment to\ncombining the statistical results of studies collected from the literature.\nWith almost any reasonable quantitative model for publication bias, only a\nsmall number of studies lost in the file-drawer will produce a significant\nbias. This result contradicts the well known Fail Safe File Drawer (FSFD)\nmethod for setting limits on the potential harm of publication bias, widely\nused in social, medical and psychic research. This method incorrectly treats\nthe file drawer as unbiased, and almost always misestimates the seriousness of\npublication bias. A large body of not only psychic research, but medical and\nsocial science studies, has mistakenly relied on this method to validate\nclaimed discoveries. Statistical combination can be trusted only if it is known\nwith certainty that all studies that have been carried out are included. Such\ncertainty is virtually impossible to achieve in literature surveys.",
"arxiv_id": "physics/9909033",
"authors": [
"Jeffrey D. Scargle"
],
"categories": [
"physics.data-an"
],
"title": "Publication Bias (The \"File-Drawer Problem\") in Scientific Inference",
"url": "https://arxiv.org/abs/physics/9909033"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "55ce7182-0108-4b67-848d-4d8a536839eb",
"id": "arXiv Dataset IDs",
"type": "Model",
"variant": "snapshot-2026-03-01",
"version": "0.1.0"
},
"user_id": 1000002
}