Premium
A Rubric for the Detection of Students in Crisis
Author(s) -
Burkhardt Amy,
Lottridge Susan,
Woolf Sherri
Publication year - 2020
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12410
Subject(s) - rubric , harm , computer science , process (computing) , classifier (uml) , artificial intelligence , machine learning , context (archaeology) , task (project management) , data science , natural language processing , mathematics education , psychology , engineering , social psychology , programming language , paleontology , systems engineering , biology
For some students, standardized tests serve as a conduit to disclose sensitive issues of harm or distress that may otherwise go unreported. By detecting this writing, known as crisis papers , testing programs have a unique opportunity to assist in mitigating the risk of harm to these students. The use of machine learning to automatically detect such writing is necessary in the context of online tests and automated scoring. To achieve a detection system that is accurate, humans must first consistently label the data that are used to train the model. This paper argues that the existing guidelines are not sufficient for this task and proposes a three‐level rubric to guide the collection of the training data. In showcasing the fundamental machine learning procedures for creating an automatic text classification system, the following evidence emerges as support of the operational use of this rubric. First, hand‐scorers largely agree with one another in assigning labels to text according to the rubric. Additionally, when this labeled data are used to train a baseline classifier, the model exhibits promising performance. Recommendations are made for improving the hand‐scoring training process, with the ultimate goal of quickly and accurately assisting students in crisis.