Evaluating the Evidence

Health outcomes research applies a wide range of methods to identify optimal approaches to determine optimal approaches to determine the effects of healthcare interventions and policies. Critical evaluation skills are necessary to navigate the evidence and identify studies that should influence clinical decision making and policy. A hierarchical “pyramid of evidence” that emphasizes randomized controlled clinical trials (RCTs) has been promulgated as the approach to judging study design and quality. Whereas this hierarchy may be suitable for the evaluation of efficacy, it may be inappropriate for many health outcomes research questions. This article examines the relevance of this construct to questions beyond those of therapeutic efficacy and discusses the essential role of study designs beyond RCTs. The strengths and weaknesses of study designs commonly encountered in the medical literature are reviewed. The article concludes with 2 case studies that apply the concepts reviewed and illustrate the need to match the study design with the research question. Of note, the concepts of internal validity, external validity, confounding, and bias are central to the discussion of the strengths and weaknesses of study designs in the medical literature. Readers not familiar with these concepts should refer to the Appendix in the online-only Data Supplement for clarification. The Pyramid of Evidence: A Useful Construct?Traditionally, the design of a study has been considered a principal barometer of the validity of its findings. In this construct, different study designs are considered in the context of a pyramid of evidence, in which studies most susceptible to threats to internal validity reside at the bottom and those least prone reside at the top (Figure). This hierarchy is widely used in discussions about the quality of medical studies and is integrated into the grading of evidence in practice guidelines.1 However, although the pyramid of evidence is undoubtedly well suited for questions of therapeutic efficacy, …