Developing a tool for observing group critical thinking skills in first-year medical students: a pilot study using physiology-based, high-fidelity patient simulations

Most assessments of physiology in medical school use multiple choice tests that may not provide information about a student's critical thinking (CT) process. There are limited performance assessments, but high-fidelity patient simulations (HFPS) may be a feasible platform. The purpose of this pilot study was to determine whether a group's CT process could be observed over a series of HFPS. An instrument [Critical Thinking Skills Rating Instrument CTSRI)] was designed with the IDEAS framework. Fifteen groups of students participated in three HFPS that consisted of a basic knowledge quiz and introduction, HFPS session, and debriefing. HFPS were video recorded, and two raters reviewed and scored all HFPS encounters with the CTSRI independently. Interrater analysis suggested good reliability. There was a correlation between basic knowledge scores and three of the six observations on the CTSRI providing support for construct validity. The median CT ratings significantly increased for all observations between the groups' first and last simulation. However, there were still large percentages of video ratings that indicated students needed substantial prompting during the HFPS. The data from this pilot study suggest that it is feasible to observe CT skills in HFPS using the CTSRI. Based on the findings from this study, we strongly recommend that first-year medical students be competent in basic knowledge of the relevant physiology of the HFPS before participating, to minimize the risk of a poor learning experience.