Premium
Managing What We Can Measure: Quantifying the Susceptibility of Automated Scoring Systems to Gaming Behavior
Author(s) -
Higgins Derrick,
Heilman Michael
Publication year - 2014
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12036
Subject(s) - computer science , measure (data warehouse) , construct (python library) , scoring system , open source , data science , risk analysis (engineering) , data mining , software , medicine , surgery , programming language
As methods for automated scoring of constructed‐response items become more widely adopted in state assessments, and are used in more consequential operational configurations, it is critical that their susceptibility to gaming behavior be investigated and managed. This article provides a review of research relevant to how construct‐irrelevant response behavior may affect automated constructed‐response scoring, and aims to address a gap in that literature: the need to assess the degree of risk before operational launch. A general framework is proposed for evaluating susceptibility to gaming, and an initial empirical demonstration is presented using the open‐source short‐answer scoring engines from the Automated Student Assessment Prize (ASAP) Challenge.