Premium
Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study
Author(s) -
Mahlke Jana,
Schultze Martin,
Eid Michael
Publication year - 2019
Publication title -
british journal of mathematical and statistical psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.157
H-Index - 51
eISSN - 2044-8317
pISSN - 0007-1102
DOI - 10.1111/bmsp.12149
Subject(s) - structural equation modeling , statistics , sample size determination , statistic , sample (material) , reliability (semiconductor) , monte carlo method , residual , econometrics , mathematics , multilevel model , standard error , computer science , algorithm , power (physics) , chemistry , physics , chromatography , quantum mechanics
When multisource feedback instruments, for example, 360‐degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non‐standard multilevel structural equation model that incorporates self‐ratings (level‐2 variables) and others’ ratings from different additional perspectives (level‐1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self‐ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self‐ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade‐off effects between the sample sizes of self‐ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ 2 test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level‐specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.