CONTEXT BIAS IN THE TEST OF ENGLISH AS A FOREIGN LANGUAGE

This study was undertaken to test the hypothesis advanced by Traynor (1985) that items of the TOEFL test that contain references to American people, places, regions, customs, institutions, etc., tend to favor examinees who have spent some time living in the United States. Two samples of examinees were drawn from the March 1987 administration of the test–one, tested in the United States, consisting of individuals who had lived in this country for more than a year, the other, tested in their native countries, consisting of individuals who had spent less than a month in the United States. Mantel‐Haenszel (1959) analyses were carried out for each of the 146 operational items of the test, using data from the two groups, separately by region of origin, and also combined across regions. In all analyses, the subjects in the two groups were matched on the total score on the section in which the item was contained. In a separate part of the study, five raters were engaged and asked to rate the items of the test, judging whether they contained explicit reference to some aspect of Americana. These ratings were found to be highly reliable; all but 16 of the 146 were unanimously judged by the raters as having reference or not having reference to Americana. Of all 146 items in TOEFL, only one gave a consistent advantage, found in every one of the five regions of the world studied here, to examineees tested domestically. It is noteworthy, however, that this item made no reference to Americana. Several other items also showed an advantage to domestic candidates, but, with two exceptions, the advantage was found in only one region of the world. The two exceptions showed a domestic advantage in only two regions. Of the five most “significantly” aberrant items, only one was judged (by four of the five raters) to be an “Americana” item. Bivariate distributions of “Americana” scores on the items of TOEFL versus their Mantel‐Haenszel indices were prepared to determine whether there was any relation between the two. None was found. On the strength of the information provided in the tables of this report, and on the strength of the review of the items cited here, it may be concluded that there is no support for the hypothesis that TOEFL items that make reference to American people, places, institutions, customs, etc., tend to advantage TOEFL candidates who have lived in the United States for a year or more over those who have spent little (one month or less) time in the country.