Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment Courses

Written formative assessments can provide instructors with rich insight into students’ thinking about scientific concepts. However, the time and effort involved in grading deter instructors from having students write in large courses. As large-enrollment introductory STEM courses become increasingly common, the need for innovations that facilitate the use of written assessments continues to grow. We piloted the use of automated text analysis to overcome these obstacles and facilitate the use of written formative assessment in a large-enrollment introductory biology course. Student responses to online homework on thermodynamics, metabolism, central dogma (genetics) and acid-base chemistry were collected in three 300+-person course sections. We used automated text analysis to extract and categorize concepts from student writing. Then, we used k-means cluster analysis to aggregate responses into distinct groups. From these analyses, we created feedback reports to provide instructors with an assessment of students’ responses before the next class period (less than one working day), so that instructors could use this feedback to inform their instruction. We present the results of this pilot study, including a description of the feedback reports and faculty instruction in response to feedback on student writing. We also describe lessons learned to improve the use of written assessments, automated analysis, rapid feedback reports and instruction in large enrollment courses. Finally, we suggest some future directions for research based on our analysis of student writing.