Poisson Regressions: A Little Fishy

When analyzing count data (such as number of questions answered correctly), psychologists often use Poisson regressions. We show through simulations that violating the assumptions of a Poisson distribution even slightly can lead to false positive rates more than doubling, and illustrate this issue with a study that finds a clearly spurious but highly significant connection between seeing the color blue and eating fish candies. In additional simulations we test alternative methods for analyzing count-data and show that these generally do not suffer from the same inflated false positive rate, nor do they result in much higher false negatives in situations where Poisson would be appropriate.