Premium
Multiple Comparison Procedures—Cutting the Gordian Knot
Author(s) -
Saville David J.
Publication year - 2015
Publication title -
agronomy journal
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.752
H-Index - 131
eISSN - 1435-0645
pISSN - 0002-1962
DOI - 10.2134/agronj2012.0394
Subject(s) - knot (papermaking) , pairwise comparison , mathematics , null hypothesis , statistics , multiple comparisons problem , statistical hypothesis testing , algorithm , calculus (dental) , computer science , engineering , medicine , dentistry , chemical engineering
Multiple comparison procedures (MCPs), or mean separation tests, have been the subject of great controversy since the 1950s. Essentially, these procedures are an attempt at simultaneously formulating and testing pairwise comparison hypotheses using data from a single experiment. An unacceptable operating characteristic of most MCPs is their “inconsistency,” an idea that is illustrated in this article. This characteristic led to the development of a “practical solution” to the MCP problem, which is to “cut the Gordian knot” by abandoning any attempt at simultaneous formulation and testing. Instead, I recommend using the simplest multiple comparison procedure, the unrestricted least significant difference procedure, to (i) formulate new hypotheses at a known “false discovery rate” (in the null case) such as 5%, and (ii) independently test interesting new hypotheses in a second experiment. I also discuss the implications for sample size calculations of the choice of MCP.