Premium
Rubric Rating with MFRM versus Randomly Distributed Comparative Judgment: A Comparison of Two Approaches to Second‐Language Writing Assessment
Author(s) -
Sims Maureen E.,
Cox Troy L.,
Eckstein Grant T.,
Hartshorn K. James,
Wilcox Matthew P.,
Hart Judson M.
Publication year - 2020
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12329
Subject(s) - rubric , writing assessment , rasch model , context (archaeology) , reliability (semiconductor) , computer science , raw score , inter rater reliability , psychology , natural language processing , rating scale , applied psychology , mathematics education , raw data , quantum mechanics , biology , programming language , paleontology , developmental psychology , power (physics) , physics
The purpose of this study is to explore the reliability of a potentially more practical approach to direct writing assessment in the context of ESL writing. Traditional rubric rating (RR) is a common yet resource‐intensive evaluation practice when performed reliably. This study compared the traditional rubric model of ESL writing assessment and many‐facet Rasch modeling (MFRM) to comparative judgment (CJ), the new approach, which shows promising results in terms of reliability. We employed two groups of raters—novice and experienced—and used essays that had been previously double‐rated, analyzed with MFRM, and selected with fit statistics. We compared the results of the novice and experienced groups against the initial ratings using raw scores, MFRM, and a modern form of CJ—randomly distributed comparative judgment (RDCJ). Results showed that the CJ approach, though not appropriate for all contexts, can be as reliable as RR while showing promise as a more practical approach. Additionally, CJ is easily transferable to novel assessment tasks while still providing context‐specific scores. Results from this study will not only inform future studies but can help guide ESL programs in selecting a rating model best suited to their specific needs.