Premium
Integrating the Analysis of Mental Operations Into Multilevel Models to Validate an Assessment of Higher Education Students’ Competency in Business and Economics
Author(s) -
Brückner Sebastian,
Pellegrino James W.
Publication year - 2016
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/jedm.12113
Subject(s) - credibility , multilevel model , psychology , argument (complex analysis) , field (mathematics) , qualitative property , scale (ratio) , item response theory , protocol analysis , computer science , applied psychology , mathematics education , psychometrics , machine learning , developmental psychology , mathematics , biochemistry , chemistry , physics , cognitive science , quantum mechanics , political science , pure mathematics , law
The Standards for Educational and Psychological Testing indicate that validation of assessments should include analyses of participants’ response processes. However, such analyses typically are conducted only to supplement quantitative field studies with qualitative data, and seldom are such data connected to quantitative data on student or item performance. This paper presents an example of how data from an analysis of mental operations collected using a sociocognitive approach can be quantitatively integrated with other data on student and item performance to validate in part an assessment of higher education students’ competency in business and economics. Evidence of forward reasoning and paraphrasing as mental operations is obtained using the think‐aloud method. As part of the validity argument and to enhance credibility of the findings, the generalized linear models are expressed as multilevel models in which the analyses of response processes are aligned with quantitative findings from large‐scale field studies.