Premium
Exploring the Influence of Range Restrictions on Connectivity in Sparse Assessment Networks: An Illustration and Exploration Within the Context of Classroom Observations
Author(s) -
Wind Stefanie A.,
Jones Eli
Publication year - 2018
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/jedm.12173
Subject(s) - context (archaeology) , principal (computer security) , range (aeronautics) , scale (ratio) , rating scale , psychology , computer science , principal component analysis , applied psychology , artificial intelligence , developmental psychology , paleontology , physics , quantum mechanics , biology , operating system , materials science , composite material
Range restrictions, or raters’ tendency to limit their ratings to a subset of available rating scale categories, are well documented in large‐scale teacher evaluation systems based on principal observations. When these restrictions occur, the ratings observed during operational teacher evaluations are limited to a subset of the available categories. However, range restrictions are less common within teacher performances that are used to establish links (anchor ratings) in otherwise disconnected assessment systems. As a result, principals’ category use may be different between anchor ratings and operational ratings. The purpose of this study is to explore the consequences of discrepancies in rating scale category use across operational and anchor ratings within the context of teacher evaluation systems based on principal observations. First, we used real data to illustrate the presence of range restriction in operational ratings, and the effect of this restriction on connectivity. Then, we used simulated data to explore these effects using experimental manipulation. Results suggested that discrepancies in range restriction between anchor and operational ratings do not systematically impact the precision of teacher, principal, and teaching practice estimates. We discuss the implications of these results in terms of research and practice for teacher evaluation systems.