Open AccessA PRELIMINARY STUDY OF RATERS FOR THE TEST OF SPOKEN ENGLISH
Author(s) -
Bejar Isaac I.
Publication year - 1985
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2330-8516.1985.tb00090.x
Subject(s) - psychology , test (biology) , index (typography) , quality (philosophy) , applied psychology , statistics , social psychology , clinical psychology , computer science , mathematics , paleontology , philosophy , epistemology , world wide web , biology
The investigation was undertaken to provide information about the feasibility of reducing scoring costs by using one rater instead of the two that are now used for the TSE. It was concluded that because of the possibility of different standards among potential raters, it does not appear feasible to use a single rater as the sole determiner of speaking proficiency under the current system. Other possible alternatives to a single rating, relying on psychometric methodology and technology, are discussed. The approach was to first examine the possibility of developing a “quality control” index that would predict the extent of the disagreement between two raters. The index that was developed for this purpose could not be validated. It was found that the best predictors of rater disagreement were the identities of the raters. The disagreements, however, resulted from the differing standards used by different raters. That is, raters agree substantially about the ordering of examinees but vary slightly in the severity of their ratings.