Analyzing Changes in the Individual Dimensions of a Behaviorally Anchored Rating Scale

Individual Item or Dimension Analysis (IDA) using the Social Relations Model (SRM) has the power to reveal subtle changes in behavior as recorded in a behaviorally anchored scale. These changes cannot be seen in Across Dimension Analysis (ADA). Analysis of mean and standard deviation across all 5 CATME dimensions does not uncover differences at the individual item (or dimension) basis. The purpose of this analysis is to establish the usefulness of IDA SRM as a statistical analysis method. The Social Relations Model decomposes variance into components. In our case, the variance is identified as rater, target, or relationship variances. Rater variance measures how individuals tend to rate their teammates. A large rater variance would indicate that students give their teammates ratings that are idiosyncratic – they don’t relate to the ratings given by others. Target variance measures how consistently individuals are rated by their teammates. A larger target effect would indicate that students tend to receive similar ratings from various raters. Relationship variance measures how ratings are affected by unique dyadic relationships after other variances have been removed. A larger relationship effect would indicate that the ratings are based more on distinct relationship pairings within the team – that is, that the various team members display different behaviors to each of their teammates and are rated based on those behaviors. The individual dimensions can be defined as follows: Having (H) relevant KSAs, Contributing (C) to the team’s work, Interacting (I) with team members, Keeping (K) the team on track, and Expecting (E) quality. In this analysis, we examined three different teamwork studies that were based on evaluating average changes in multiple behaviorally anchored items and found additional insights into behavior changes through IDA. All data included in this analysis was obtained through the CATME peer evaluation system and our studies focused on methods of improving the quality of peer evaluations. In an initial study of a large Midwestern university, IDA SRM analysis revealed significant results in dimensions C, I, K, and H that were not seen using an ADA analysis, the average rating scores across items (or dimensions). Our IDA analyses will compare IDA and ADA analyses and intervention groups who received various types of teamwork training and control groups who did not receive any teamwork training.