In praise of critical appraisal

One of the advantages of a multidisciplinary field like public health is that the good practices developed by one discipline can be shared with other disciplines. Critical appraisal of the medical literature is a basic procedure in evidence-based medicine. We think it has benefit for other researchers, practitioners and policy makers. The idea of critique is familiar to social scientists, especially those interested in critical social theory. Sometimes critical theorists are, in turn, criticised for always trying to swim against the tide, but their aim is to raise questions about taken-for-granted realities in order to gain a different, better understanding of social life. Clinicians are used to ‘grand rounds’ or case conferences in which colleagues from various sub-specialties gather to hear details of a specific patient’s illness. The case presenter takes the lead, but the aim is to have different views of the illness presented and discussed, with differences being resolved in the most positive way possible. With the growth in the medical literature since World War II, clinicians had to engage with the medical literature in a new way. If it is not to overwhelm, the literature has to be brought under control. Not only must the literature be read, but it must be critically assessed to determine its relevance for practice. There are now excellent users’ guides to the medical literature, published as a series in the Journal of the American Medical Association (JAMA) and collected in book form. Australian researchers have made substantial contributions to the work of critical appraisal . In public health we can draw on these (and other) traditions in setting in place journal clubs of people who gather with the purpose of critically appraising selected articles from the literature. The rules for critical appraisal of the clinical medical literature are explicit and readily accessible but are not necessarily applicable to other disciplines, using other, different research methods. We suggest that the following questions are worth asking of any article presented for discussion in a journal club. 1. Is this a significant research problem? There are clearly some research questions that are trivial and not worth pursuing or publishing. A thorough review of the literature, both local and international, serves to establish the importance of the research problem and shows that the answers have not yet been generated by other researchers. 2. Has the appropriate research design been used? Different research designs are suitable for different problems and an appropriate research design is the one that is best adapted to answering the research problem. Specific research methods from randomised controlled trials to qualitative research methods are better suited for addressing specific kinds of problems and it is the problem rather than the disciplinary preferences of the research team that should determine the method used. 3. Has the method been used correctly? We need a clear description of the methods that have been used and how they were applied to the research problem. There is clear guidance for what is to count as a methodologically refined study in most disciplines, for most methods, and this allows us to assess the methodological basis of the study. 4. How does context affect the research design? As any researcher knows, there are research contexts in which the perfect research design is neither practical nor even desirable. Among these are vulnerable populations that may be especially difficult to enrol in potentially intrusive studies. The research design may need to be adapted to one that is perhaps less than ideal from a methodological perspective alone. The test is whether the design has been implemented as well as possible, given the contextual constraints that should be outlined in the article. The UK ‘Sure Start’ program – an intervention in disadvantaged areas – explicitly excluded randomisation of communities. The outcome was that the interpretation of the findings, showing no benefit and some evidence of harm, was questionable. 5. Are the results trustworthy? A good article with results that can readily be implemented in practice or in policy making is one where the method is used to produce data that are analysed to produce a set of defensible conclusions, and where each of these steps is clearly described and justified. In many research studies there are limitations to the study – which can still make an important contribution to a field of knowledge – but these limitations have to be identified and the possible implications for the conclusions should be addressed. The point of the critical appraisal exercise is to assess research studies in the public health literature for their potential contribution to practice or policy. In public health, critical appraisal exercises provide a good setting for different disciplines to learn about the quality criteria and methodological niceties of their colleagues trained in other disciplines. In time we can develop a better, shared, overall understanding of public health research method. This is, however, not just an exercise designed for assessing the literature. Researchers learn new skills from critical appraisal exercises. In disciplines where there are not as yet clear criteria for assessing the quality of a research report, it helps to focus the debate on how we are to judge research quality. We learn skills in communicating the details of research design in articles submitted for publication. We may even learn to design our research studies more carefully so that we are able to reach the most defensible and valuable conclusions, within the constraints that all researchers experience ranging from inaccessibility of research participants to lack of funding for a full-scale study.