Premium
Impact of choice of quality appraisal tool for systematic reviews in overviews
Author(s) -
Pieper Dawid,
Mathes Tim,
Eikermann Michaela
Publication year - 2014
Publication title -
journal of evidence‐based medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.885
H-Index - 22
ISSN - 1756-5391
DOI - 10.1111/jebm.12097
Subject(s) - critical appraisal , systematic review , psychology , reliability (semiconductor) , context (archaeology) , scale (ratio) , interpretability , quality (philosophy) , rating scale , applied psychology , management science , medline , computer science , medicine , alternative medicine , pathology , artificial intelligence , biology , developmental psychology , quantum mechanics , economics , paleontology , biochemistry , philosophy , epistemology , power (physics) , physics
Objective The question whether the choice of a critical appraisal tool has an impact on the result of the evidence synthesis in systematic reviews has been neglected by research. This is also true for psychometric properties of critical appraisal tools. The objective of the study is to exemplify that in the context of overviews (reviews of reviews). Methods Based on a published overview investigating the hospital volume‐outcome relationship in surgery, 32 therein included systematic reviews were independently evaluated with four critical appraisal tools by two reviewers. We rated the relationship on a five‐point rating scale using qualitative evidence synthesis. Measures of reliability and correlation coefficients were calculated. Results The result of the evidence synthesis was not dependent on the choice of a critical appraisal tool. Inter‐rater reliability differed depending on the tool, Cohens Kappa ranging from 0.47 to 0.76. There was a high heterogeneity between the two pairs of reviewers. Conclusion The choice of a critical appraisal tool has no impact on the result of the evidence synthesis, despite differences in the covered components by each CAT. Further studies should concentrate on investigating psychometric properties and the impact of choice of CATs on the evidence synthesis in other contexts. The high heterogeneity between the two pairs of reviewers, all of them experienced in appraising systematic reviews, indicates a degree of interpretability in the items.