Not At Issue

There is lots of truth in what Professor McGorry says.1 We are not “at issue.” Policymakers often have to (and should) go well beyond evidence as presented in a review of trials or, for that matter, reviews of other methodologies. They have to consider not only best evidence but also resource, public requirements, and opinion of experts to come to some workable decision—or defer.2–4 There were certain issues that might help to be clarified. Cochrane reviews are different in many ways to either traditional reviews (broadly speaking, opinion pieces without any ‘Methods’ section) or even standard systematically conducted reviews (those with a workable ‘Methods’ section published on article). Cochrane reviewers try to use objective methodology to summarize best evidence. This is an attempt at fairly summarizing fair tests of different treatment approaches—even complex interventions for complex conditions. This can be difficult and opinion does creep in. Where subjectivity is involved, however, this should be made explicit. For example, the 2 excluded trials referred to in the companion piece as “positive” are excluded from the Cochrane review for explicit reasons. (The full details are not published on the Cochrane Corner but are available on the Cochrane Library.) Now, those reasons may or may not be correct in the opinion of the reader—but the reasons for exclusion are there to see and can be considered and, if indicated, challenged. Here, Cochrane reviews differ again. They are maintained—updated in light of new evidence or valid criticism—so different from those fossilized in article print. Those interested in this area need only press the “Submit comments” button on the Cochrane Library to have direct input into a review and for their comments to be published alongside the review. Rather than investing effort in publicizing disquiet those interested in the area can make constructive criticism in order to improve the end product. The review can be the best available because of a genuinely collaborative effort of part of the subspeciality—again setting this type of work apart from much that goes into article format. This improvement, however, does have to be within the confines of what Cochrane does. The unreferenced suggestion in the companion piece1 that Cochrane follows orthodoxy is, in some ways, correct. Probably, the best citation for this orthodoxy is the Cochrane Handbook5 containing, for example, explicit guidance of how to identify relevant studies and incorporate data from both randomized trials and how to include data from nonrandomized studies (chapter 13). It is a common misconception that Cochrane reviews are confined to just randomized trials. Cochrane reviewers are, however, confined to being systematic. Including data from studies that are not randomized becomes ever more prone to many biases. For randomized trials, we have a much better idea of relevant biases and quantification of their influences. Some things, however, are truly impossible to randomize, and others are unethical. Conversely, recognizing the randomized trial still to be the gold standard of fair testing, pioneers in early intervention have seen that implementation is unethical divorced from trial data. The Cochrane review tries to summarize these (largely) trial-derived data. Thereafter just how these data are then used—or not—is quite another matter. There has long been the concern that Cochrane reviews are used to fuel those who wish to pursue policy dictation by numbers6—a practice no better than dictation by preformed opinion. Do not shoot the (Cochrane) messenger. Traditional and systematic reviews have to both be considered by policymakers. Often both concur. For example, psychiatry did not need a Cochrane review to know chlorpromazine was effective7 but it did to answer the patient’s direct question “What is my odds of really getting better on this drug?” or the policymakers “How many people have to be given this treatment in order to avoid costly outcomes?”8 Sometimes, however, the opinion of experts radically differs from objective synthesis of best evidence. Researchers can advocate clinical direction quite opposite to that indicated by systematic summaries of best evidence.9,10 Countless resource can be wasted. It is not difficult to see how the company producing a traditional review of the effects of their new drug would find it difficult give an unbiased account of their product. In the same way, pioneers in truly moving forward the care of people with schizophrenia with their vision and passion, and science may not be the best choice for dispassionate appraisal. There is a place for both the informed expert opinion and scientific appraisal of best evidence—with a clear workable definition of the latter. Policymakers need both perspectives. Their role, recommending a pathway of care, is not an enviable one. It is hard to please all of the people all of the time.