Open AccessThe Impact of Aberrant Responses and Detection in Forced‐Choice Noncognitive Assessment
Author(s) -
Kim Sooyeon,
Moses Tim
Publication year - 2018
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/ets2.12222
Subject(s) - two alternative forced choice , statistic , pairwise comparison , outlier , trait , statistics , psychology , preference , range (aeronautics) , dimension (graph theory) , function (biology) , item response theory , econometrics , social psychology , mathematics , computer science , psychometrics , materials science , evolutionary biology , pure mathematics , composite material , biology , programming language
The purpose of this study is to assess the impact of aberrant responses on the estimation accuracy in forced‐choice format assessments. To that end, a wide range of aberrant response behaviors (e.g., fake, random, or mechanical responses) affecting upward of 20%–30% of the responses was manipulated under the multi‐unidimensional pairwise preference (MUPP) model with forced‐choice format. Under each aberrance condition, we also computed the lz statistic (a common person‐fit index) to determine how well this index can accurately detect aberrant responses without erroneously misidentifying honest examinees. Some aberrant behaviors (e.g., limited responses) were more problematic than others (e.g., omitting) in terms of true trait level recovery. The lz statistic associated with each dimension of the MUPP form led to higher detection rates than did the overall lz statistic. When the dimensional lz approach was used, however, many honest examinees were misclassified as outliers. The detection rates of lz differed as a function of aberrant response types.