Premium
Generalizability Theory as Evidence of Concerns About Fairness in Large‐Scale ESL Writing Assessments
Author(s) -
HUANG JINYAN
Publication year - 2011
Publication title -
tesol journal
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.468
H-Index - 14
eISSN - 1949-3533
pISSN - 1056-7941
DOI - 10.5054/tj.2011.269751
Subject(s) - generalizability theory , rating scale , reliability (semiconductor) , psychology , variation (astronomy) , scale (ratio) , inter rater reliability , mathematics education , developmental psychology , geography , power (physics) , physics , cartography , quantum mechanics , astrophysics
Using generalizability theory, this study examined both the rating variability and reliability of English as a second language (ESL) students' writing in two provincial examinations in Canada. This article discusses expected and unexpected similarities and differences related to rating variability and reliability between the two testing programs. As expected, there was more desired but less unwanted variation in ESL and native‐English‐speaking (NES) students' writing scores in Province B than in Province A. But unexpectedly, the results demonstrated systematic differences between ESL and NES students in terms of rating variability. Further, there were lower reliabilities in ESL students' scores in comparison to NES students' scores. These findings raise potential concerns about the fairness of large‐scale ESL writing assessments.