
Is Differential Noneffortful Responding Associated With Type I Error in Measurement Invariance Testing?
Author(s) -
Joseph A. Rios
Publication year - 2021
Publication title -
educational and psychological measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.819
H-Index - 95
eISSN - 1552-3888
pISSN - 0013-1644
DOI - 10.1177/0013164421990429
Subject(s) - measurement invariance , differential item functioning , equivalence (formal languages) , statistics , metric (unit) , mathematics , context (archaeology) , differential (mechanical device) , scalar (mathematics) , psychology , social psychology , econometrics , item response theory , confirmatory factor analysis , psychometrics , pure mathematics , physics , structural equation modeling , paleontology , operations management , thermodynamics , geometry , economics , biology
Low test-taking effort as a validity threat is common when examinees perceive an assessment context to have minimal personal value. Prior research has shown that in such contexts, subgroups may differ in their effort, which raises two concerns when making subgroup mean comparisons. First, it is unclear how differential effort could influence evaluations of scale property equivalence. Second, if attaining full scalar invariance, the degree to which differential effort can bias subgroup mean comparisons is unknown. To address these issues, a simulation study was conducted to examine the influence of differential noneffortful responding (NER) on evaluations of measurement invariance and latent mean comparisons. Results showed that as differential rates of NER grew, increased Type I errors of measurement invariance were observed only at the metric invariance level, while no negative effects were apparent for configural or scalar invariance. When full scalar invariance was correctly attained, differential NER led to bias of mean score comparisons as large as 0.18 standard deviations with a differential NER rate of 7%. These findings suggest that test users should evaluate and document potential differential NER prior to both conducting measurement quality analyses and reporting disaggregated subgroup mean performance.