Setting school‐level outcome standards

Background  To establish international standards for medical schools, an appropriate panel of experts must decide on performance standards. A pilot test of such standards was set in the context of a multidimensional (multiple‐choice question examination, objective structured clinical examination, faculty observation) examination at 8 leading schools in China. Methods  A group of 16 medical education leaders from a broad array of countries met over a 3‐day period. These individuals considered competency domains, examination items, and the percentage of students who could fall below a cut‐off score if the school was still to be considered as meeting competencies. This 2‐step process started with a discussion of the borderline school and the relative difficulty of a borderline school in achieving acceptable standards in a given competency domain. Committee members then estimated the percentage of students falling below the standard that is tolerable at a borderline school and were allowed to revise their ratings after viewing pilot data. Results  Tolerable failure rates ranged from 10% to 26% across competency domains and examination types. As with other standard‐setting exercises, standard deviations from initial to final estimates of the tolerable failure rates fell, but the cut‐off scores did not change significantly. Final, but not initial cut‐off scores were correlated with student failure rates ( r =  0.59, P  = 0.03). Discussion  This paper describes a method to set school‐level outcome standards at an international level based on prior established standard‐setting methods. Further refinement of this process and validation using other examinations in other countries will be needed to achieve accurate international standards.