Efforts to Improve Undergraduate Grader Consistency: A Qualitative Analysis

In this research paper, we explore the difficult decisions faced by large-scale, multisection courses in early undergraduate engineering education regarding fair and consistent assessment of student learning across sections. Our previous analysis of grading patterns of undergraduate graders in a first-year engineering course revealed that divergent decisions likely stemmed from two sources: insufficient grader training and ambiguities in rubrics and assignments. After revising rubrics and implementing grader training for a semester, we conducted think-aloud interviews with 17 undergraduate graders regarding grading, rubrics, and training. Qualitative analysis identified four technical aspects of rubrics that led to divergent grading decisions (wordiness, redundancies, unexpected solutions, and grade misfit) and five aspects that limited training effectiveness (length, misalignment, insufficient feedback, limited consequences, and philosophical misunderstanding). These findings contribute nuance to and extend upon aspects of rubric design and undergraduate grader training that have been previously identified in the literature. Recommendations related to issues identified are provided.