Premium
A Critical Review of Some Qualitative Research Methods Used to Explore Rater Cognition
Author(s) -
Suto Irenka
Publication year - 2012
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/j.1745-3992.2012.00240.x
Subject(s) - repertory grid , strengths and weaknesses , think aloud protocol , cognition , computer science , task (project management) , psychology , recall , process (computing) , qualitative research , subjectivity , applied psychology , management science , data science , cognitive psychology , social psychology , usability , human–computer interaction , social science , neuroscience , sociology , operating system , philosophy , management , epistemology , economics
Internationally, many assessment systems rely predominantly on human raters to score examinations. Arguably, this facilitates the assessment of multiple sophisticated educational constructs, strengthening assessment validity. It can introduce subjectivity into the scoring process, however, engendering threats to accuracy. The present objectives are to examine some key qualitative data collection methods used internationally to research this potential trade‐off, and to consider some theoretical contexts within which the methods are usable. Self‐report methods such as Kelly's Repertory Grid, think aloud, stimulated recall, and the NASA task load index have yielded important insights into the competencies needed for scoring expertise, as well as the sequences of mental activity that scoring typically involves. Examples of new data and of recent studies are used to illustrate these methods’ strengths and weaknesses. This investigation has significance for assessment designers, developers and administrators. It may inform decisions on the methods’ applicability in American and other rater cognition research contexts.