MONTE CARLO BASELINES FOR INTERRATER RELIABILITY CORRELATIONS USING THE POSITION ANALYSIS QUESTIONNAIRE

The Position Analysis Questionnaire (PAQ; McCormick, Jeanneret, & Mecham, 1972) has usually demonstrated acceptable reliability: When measured as the average interrater correlation, reliabilities of .80 and above are common. In contrast, recent attempts to reduce the cost and intrusiveness of PAQ job analysis by providing raters with minimal job descriptive information have reported reliabilities under .50. Because many PAQ items are clearly irrelevant for any given job–even with only minimal job information–the fact that raters agree that these items receive a “Does Not Apply” (DNA) rating may mask disagreement on the remaining items and overestimate interrater reliability, making it difficult to interpret these reliabilities. To facilitate the interpretation of PAQ interrater reliabilities, we performed a Monte Carlo study that varied the number of raters as well as the number of DNA agreements. To estimate the worst possible reliability that could be expected for a given DNA agreement level, the remaining non‐DNA ratings were random numbers. Our results showed that reliabilities in the 50 range (such as were reported in recent nontraditional PAQ studies) can be obtained when raters rule out only 15–20% of the items as DNA and respond randomly to the remainder. Implications of these results for future job analysis research and practice are discussed.