Premium
Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors
Author(s) -
McCaffrey Daniel F.,
Yuan Kun,
Savitsky Terrance D.,
Lockwood J. R.,
Edelen Maria O.
Publication year - 2014
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12061
Subject(s) - class (philosophy) , psychology , mathematics education , inter rater reliability , interpretation (philosophy) , dimension (graph theory) , test (biology) , factor (programming language) , classroom management , developmental psychology , computer science , mathematics , artificial intelligence , rating scale , paleontology , pure mathematics , biology , programming language
We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.