Premium
Manipulating Processing Difficulty of Reading Comprehension Questions: The Feasibility of Verbal Item Generation
Author(s) -
Gorin Joanna S.
Publication year - 2005
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/j.1745-3984.2005.00020.x
Subject(s) - comprehension , reading comprehension , affect (linguistics) , psychology , cognitive psychology , reading (process) , trait , encoding (memory) , test (biology) , cognition , generative model , natural language processing , item response theory , computer science , artificial intelligence , psychometrics , generative grammar , developmental psychology , linguistics , communication , philosophy , programming language , paleontology , neuroscience , biology
Based on a previously validated cognitive processing model of reading comprehension, this study experimentally examines potential generative components of text‐based multiple‐choice reading comprehension test questions. Previous research ( Embretson & Wetzel, 1987 ; Gorin & Embretson, 2005 ; Sheehan & Ginther, 2001 ) shows text encoding and decision processes account for significant proportions of variance in item difficulties. In the current study, Linear Logistic Latent Trait Model (LLTM; Fischer, 1973 ) parameter estimates of experimentally manipulated items are examined to further verify the impact of encoding and decision processes on item difficulty. Results show that manipulation of some passage features, such as increased use of negative wording, significantly increases item difficulty in some cases, whereas others, such as altering the order of information presentation in a passage, did not significantly affect item difficulty, but did affect reaction time. These results suggest that reliable changes in difficulty and response time through algorithmic manipulation of certain task features is feasible. However, non‐significant results for several manipulations highlight potential challenges to item generation in establishing direct links between theoretically relevant item features and individual item processing. Further examination of these relationships will be informative to item writers as well as test developers interested in the feasibility of item generation as an assessment tool.