Premium
Impact of unreliability on intraclass correlation coefficient and its implications for studies with clustered data
Author(s) -
Ma Yue,
Jonaitis Erin M,
Hulle Carol A,
Koscik Rebecca L,
Jones Richard N,
Carlsson Cynthia M.
Publication year - 2020
Publication title -
alzheimer's and dementia
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 6.713
H-Index - 118
eISSN - 1552-5279
pISSN - 1552-5260
DOI - 10.1002/alz.045550
Subject(s) - intraclass correlation , statistics , reliability (semiconductor) , sample size determination , variance (accounting) , mathematics , standard error , cluster (spacecraft) , mean squared error , cluster analysis , correlation , correlation coefficient , sample (material) , econometrics , reproducibility , computer science , chemistry , physics , geometry , power (physics) , accounting , chromatography , quantum mechanics , business , programming language
Background Studies using cerebrospinal fluid (CSF) biomarkers often combine CSF samples collected at different centers or assayed in different laboratories or batches, creating a clustered data structure. Such clustering needs to be controlled in the analysis of the combined data. Intraclass correlation coefficient (ICC), i.e., the ratio of the between‐cluster variance over the total variance, is used to quantify the magnitude of the clustering. The between‐cluster variance is a mix of (a) difference in CSF samples and (b) variation in measurement (i.e., unreliability). Thus the observed ICC (under unreliability) would deviate from the true ICC (under perfect reliability). We studied (1) how the observed ICC would deviate and (2) the statistical performance in its estimation. Method Aim (1) was achieved through mathematical derivations based on Classical Test Theory. Aim (2) was achieved using a simulation study to evaluate the relative bias, empirical standard error (se), and root mean square error (RMSE) in the sample estimates of the observed ICC, under three reliability levels (0.7, 0.8, 0.9), three true ICC values (0.1, 0.3, 0.5), and two cluster sizes (10, 50), with 10,000 replications for each condition. Result Aim (1): The observed ICC is greater than the true ICC. The inflation, represented as their ratio, is a function of reliability, true ICC, and cluster size (Figure 1). It decreases with increased reliability or true ICC, or decreased cluster size (Figures 2, 3). Aim (2): The sample estimates had very small negative bias (within 3%), which generally decreased with increased cluster size or true ICC, or reduced reliability. The empirical se and RMSE decreased with increased cluster size, but their relations with reliability or true ICC varied across conditions (Table 1). Conclusion The observed ICC is deviated (inflated), but unbiased. Using reliable measure is crucial when CSF samples have low variation between clusters to avoid substantial inflation in ICC. Increasing cluster size would exacerbate the inflation, yet would also increase estimation accuracy and efficiency. Such tradeoffs should be considered with planning CSF sample collection and assay. These findings can be applied to clustered data of other outcomes such as imaging and cognition.