Open AccessInternational Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews
Author(s) -
Inga Arffman
Publication year - 2012
Publication title -
isrn education
Language(s) - English
Resource type - Journals
ISSN - 2090-8652
DOI - 10.5402/2012/179824
Subject(s) - comparability , test (biology) , psychology , reading (process) , quality (philosophy) , linguistics , cognitive psychology , cognition , psycholinguistics , language assessment , mathematics education , paleontology , philosophy , mathematics , epistemology , combinatorics , neuroscience , biology
In international education studies, the different-language test versions need to be equally difficult to read and answer for the test to be valid. To ensure comparability, several quality control procedures have been developed. Among these, surprisingly little attention has been paid to judgmental reviews and their ability to identify language-related sources of bias. Also, the reviews have often failed in identifying biases. This paper explored whether it is possible to improve the ability of judgmental reviews to identify language-related sources of bias. A new review was made of two Finnish items which in the PISA (Programme for International Student Assessment) 2000 reading test showed differential item functioning but for which no clear language-related explanations were found in the review in 2000. The items were compared systematically, at all linguistic levels, to the corresponding items in the English and French source versions, at the same time taking into account the cognitive processes required to answer them and students’ written responses to them. Language-related explanations were found for both items which may have led to differences in performance, suggesting that it is possible to make judgmental reviews better able to identify language-related bias. Suggestions are given on how to do this.