The Meta-Distribution of Standard P-Values
Author(s) -
Nassim Nicholas Taleb
Publication year - 2016
Publication title -
ssrn electronic journal
Language(s) - English
Resource type - Journals
ISSN - 1556-5068
DOI - 10.2139/ssrn.2834266
Subject(s) - distribution (mathematics) , geography , statistics , environmental science , mathematics , mathematical analysis
We present an exact probability distribution (meta-distribution) for p-values across ensembles of statistically identical phenomena, as well as the distribution of the minimum p-value among m independents tests. We derive the distribution for small samples 2 P-values are shown to be extremely skewed and volatile, regardless of the sample size $n$, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude.The formulas allow the investigation of the stability of the reproduction of results and "p-hacking" and other aspects of meta-analysis. From a probabilistic standpoint, neither a p-value of .05 nor a "power" at .9 appear to make the slightest sense.
Accelerating Research
Robert Robinson Avenue,
Oxford Science Park, Oxford
OX4 4GP, United Kingdom
Address
John Eccles HouseRobert Robinson Avenue,
Oxford Science Park, Oxford
OX4 4GP, United Kingdom