The P Value: Probable Does Not Mean Practical

Nearly everyone is familiar with the literary detective Sherlock Holmes and his often-picked-on associate Dr. John Watson. In “A Scandal in Bohemia,” Holmes educates Watson on the difference between seeing and observing. That Watson is looking at the same information, yet deriving different (and more often incorrect) conclusions than Holmes, befuddles him. “I believe that my eyes are as good as yours,” Watson claims. To which Holmes answers “Quite so. You see, but you do not observe. The distinction is clear.”The same holds true for the interpretation of scientific data. Researchers see data, but may fail to observe whether the data yield any meaningful results. Researchers reach conclusions, but may fail to observe whether the data support the conclusions. In doing so, they may miss important clues that the data provide.Overreliance on statistical analyses of data is like seeing a snapshot of your data but not really observing the data. The P value is an excellent example of how researchers may over- or underinterpret data and thus fail to see the true picture. To illustrate this fact, let us start with what the P value really is.In research, one can never state with 100% certainty that any change or difference is real. In fact, probability testing starts with the assumption that the difference between groups is zero (the null hypothesis). Therefore, all one can do is determine the probability ( P value) that the null hypothesis is true. If the P value is small enough, it suggests, but does not prove, that the …