Premium
The ROBINS‐I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta‐analysis
Author(s) -
Zhang Yuhui,
Huang Litao,
Wang Dandan,
Ren Pengwei,
Hong Qi,
Kang Deying
Publication year - 2021
Publication title -
journal of evidence‐based medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.885
H-Index - 22
ISSN - 1756-5391
DOI - 10.1111/jebm.12427
Subject(s) - observational study , meta analysis , statistics , statistic , reliability (semiconductor) , medicine , systematic review , medline , mathematics , law , power (physics) , physics , quantum mechanics , political science
Abstract Objective There is a lack of evidence on the usage of the quality assessment tool‐the Risk Of Bias In Nonrandomized Studies‐of Interventions (ROBINS‐I). This article aimed to measure the reliability, criterion validity, and feasibility of the ROBINS‐I and the Newcastle‐Ottawa Scale (NOS). Methods A sample of systematic reviews or meta‐analyses of observational studies were selected from Medline (2013‐2017) and assessed by two reviewers using ROBINS‐I and the NOS. We reported on reliability in terms of the first‐order agreement coefficient (AC1) statistic. Correlation coefficient statistic was used to explore the criterion validity of the ROBINS‐I. We compared the feasibility of the ROBINS‐I and NOS by recording the time to complete an assessment and the instances where assessing was difficult. Results Five systematic reviews containing 41 cohort studies were finally included. Interobserver agreement on the individual domain of the ROBINS‐I as well as the NOS was substantial with a mean AC1 statistic of 0.67 (95% CI: 0.50‐0.83) and 0.73 (95% CI: 0.65‐0.81), respectively. The criterion validity of the ROBNS‐I was moderate ( K = 0.52) against NOS. The time in assessing a single study by ROBINS‐I varied from 7 hours initially to 3 hours compared with 30 minutes for the NOS. Both reviewers rated “bias due to departure from the intended interventions” the most time‐consuming domain in the ROBINS‐I, items in the NOS were equal. Conclusions The ROBINS‐I and the NOS seem to provide the same reliability but vary in applicability. The over‐complicated feature of ROBINS‐I may limit its usage and a simplified version is needed.