Counting apples with oranges: a limitation of the discrimination index

Objectives Three consecutive diets of the Professional and Linguistic Assessment Board's multiple choice question examination in Medicine were analysed in two different ways, taking into account the responses of the candidates to the three separate specialities within medicine. Design First, discrimination indices were obtained from calculations involving all 60 questions, each with five completions. Then, the 60 questions were analysed separately in the three constituent disciplines of Surgery (15 questions), Obstetrics and Gynaecology (12 questions) and Medicine (33 questions). Setting Professional and Linguistic Assessment Board. Results By the first analysis, performed by considering the whole paper to be a single entity, discrimination indices for medicine were greater than for the other two disciplines; one candidate obtained a random score, i.e. one that was no better than could have arisen, with 95% confidence, by chance as predicted by the hypergeometric distribution, and reliability coefficients were high (0·87). In the second analysis, the paper was considered as three separate components and discrimination indices calculated for each discipline separately. Discrimination indices were similar in all three disciplines, there were 26 random scores, and reliability coefficients were reduced – moderately in Medicine, considerably in the other two disciplines. Conclusions These findings emphasize the importance of defining the territory of the subject and ensuring that all the items are drawn from that territory, as well as of using a sufficiently large number of items.