Premium
Evaluating Content‐Related Validity Evidence Using a Text‐Based Machine Learning Procedure
Author(s) -
Anderson Daniel,
Rowley Brock,
Stegenga Sondra,
Irvin P. Shawn,
Rosenberg Joshua M.
Publication year - 2020
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12314
Subject(s) - categorical variable , computer science , content validity , test (biology) , congruence (geometry) , natural language processing , information retrieval , vocabulary , content analysis , content (measure theory) , artificial intelligence , representation (politics) , machine learning , psychology , statistics , psychometrics , social psychology , mathematics , linguistics , paleontology , philosophy , social science , mathematical analysis , sociology , politics , political science , law , biology
Validity evidence based on test content is critical to meaningful interpretation of test scores. Within high‐stakes testing and accountability frameworks, content‐related validity evidence is typically gathered via alignment studies, with panels of experts providing qualitative judgments on the degree to which test items align with the representative content standards. Various summary statistics are then calculated (e.g., categorical concurrence, balance of representation) to aid in decision‐making. In this paper, we propose an alternative approach for gathering content‐related validity evidence that capitalizes on the overlap in vocabulary used in test items and the corresponding content standards, which we define as textual congruence . We use a text‐based, machine learning model, specifically topic modeling, to identify clusters of related content within the standards. This model then serves as the basis from which items are evaluated. We illustrate our method by building a model from the Next Generation Science Standards , with textual congruence evaluated against items within the Oregon statewide alternate assessment. We discuss the utility of this approach as a source of triangulating and diagnostic information and show how visualizations can be used to evaluate the overall coverage of the content standards across the test items.