Commentary: Searching for trials for systematic reviews: what difference does it make?

Mike Clarke

UK Cochrane Centre, NHS R&D Programme, Summertown Pavilion, Middle Way Oxford OX2 7LG, UK. E-mail: mclarke{at}cochrane.co.uk

Systematic reviews are increasingly common as a means of assessing the relative effectiveness of health care interventions. This is not least because of the formation of the Cochrane Collaboration with its aim to prepare, maintain and promote the accessibility of reviews in all areas of health care.1 Just as systematic reviews contribute to the practice of evidence-based health care, so should the conduct of reviews be based on good evidence, ideally from empirical research. The paper by Juni et al.2 adds to this evidence base.

For all but the last century, decisions on how to treat patients were almost always based on personal experience, anecdotal case histories and highly selective retrospective comparisons; patients who had been treated in one way compared with others who had been treated differently. These processes are subject to many biases. The adoption of the randomized trial provided a more reliable way to estimate the relative effects of different interventions. However, for many conditions, the results of a single trial will rarely be sufficient. Most trials are too small, or too focused on a particular type of patient to provide a result that is either easily or reliably applicable to future patients.

To overcome this, a number of trials need to be brought together and this needs to be done reliably. It should minimize the possibility that chance will dominate the results (by combining the maximum amount of relevant information), minimize the possibility that bias will dominate the results (by combining an unbiased set of trials) and maximize the ability of users of the review to judge its applicability for specific health care decisions (by combining as wide a variety of trials as sensible and possible). Ideally, this would require that all relevant randomized trials are identified and included in the review.3 Practically, though, this might rarely be possible given the time and resources needed to identify all trials and to obtain complete information and data from them.4

Trial identification is one of the most important steps in the conduct of a systematic review. It needs to ensure that as many as possible of the relevant trials are found and, in particular, that a biased set of trials, where the bias stems from the results of the trials, is not identified. Publication bias, in which trials are more likely to be published if they have statistically significant positive results, makes this especially important. Coupling this with the fact that trials with more positive results are more likely to be published in English,5 it implies that searching must encompass publications in other languages.

Juni et al. investigated a sample of meta-analyses published in the mid-1990s to assess the effect of including trials published in languages other than English in these. They found that, on average, the exclusion of such trials would have had little effect on the summary effect estimates.2 However, as they point out, the size or direction of the effect was difficult to predict for individual reviews and they conclude that comprehensive searching is still required. I agree with this conclusion but, is it possible that they have actually underestimated the effect of trials published in languages other than English?

The meta-analyses chosen for the research had to include a minimum of five trials for sound statistical reasons. These are, as a consequence, more robust to the effects of removing one or two trials than meta-analyses with fewer trials. Thus it is difficult to know how applicable the results of this research might be for meta-analyses that include fewer than five trials. Such meta-analyses are not uncommon and it might be expected that the exclusion of, for example, one trial from a total of three could have a much larger effect.

The included trials that were published in languages other than English were typically smaller and had more positive results than the trials published in English. This might be an indication that the former were more subject to publication bias than the latter. This could result from the fact that, in general, trials reported in English were more readily available in the period when the reviews under investigation were done, because for example of the searching of electronic databases such as MEDLINE, which have a preponderance for articles published in English. This means that the trials published in languages other than English that were most likely to be found by reviewers—given the time and resource difficulties of searching extensively in the non-English-language literature—would be those that were published and cited with greatest prominence. Namely, the trials that were particularly positive and, most likely, particularly small.

This raises a very challenging question. Was it worthwhile for the reviewers in the sample studied by Juni et al. to have done what was probably a limited amount of searching for trials published in languages other than English, if this identified a biased set, albeit containing a relatively small amount of randomized evidence? Fortunately, recent developments have made this question less pertinent but it still needs to be borne in mind by the people who do reviews and the people who use them.

The two most important developments are the continuing growth in the number of Cochrane reviews and the increased ease of trial identification in recent years. Cochrane reviews are published electronically in full and can therefore be revised and updated as additional evidence comes to light. This evidence might be from new research or it might be newly found evidence from older research. Trial finding has become easier with the development of The Cochrane Controlled Trials Register in The Cochrane Library. This incorporates records for reports of randomized trials regardless of language restrictions and one of its main sources is the hand searching of thousands of journals and other sources around the world. This searching, which would be impossible within the context of an individual review, is done in such a way that each source is searched for all reports of studies that might be randomized trials and the results of this searching are then incorporated into The Cochrane Controlled Trials Register.6

Therefore, in agreeing with Juni et al. that comprehensive searching should remain an important component of systematic reviews, I also hope that reviewers of the mid-2000s will discover that it is now much easier to do this. The remaining challenge for reviewers will then be how to extract or obtain the necessary information and data from trials that have been published in languages other than their own.

Acknowledgments

Mike Clarke is Associate Director (Research) at the UK Cochrane Centre, which is part of the NHS Research and Development Programme. The views expressed in this editorial are his own and are not necessarily the views or the official policy of the Cochrane Collaboration.

References

1 Clarke M, Langhorne P. Revisiting the Cochrane Collaboration. Br Med J 2001;323: 821.[Free Full Text]

2 Juni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol 2002;31: 115–23.[Abstract/Free Full Text]

3 Clarke MJ, Stewart LA. Obtaining data from randomised controlled trials: how much do we need to perform reliable and informative meta-analyses? Br Med J 1994;309: 1007–10.[Free Full Text]

4 Egger M, Davey Smith G. Bias in location and selection of studies. Br Med J 1998;316: 61–66.[Free Full Text]

5 Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350: 326–29.[CrossRef][ISI][Medline]

6 Lefebvre C, Clarke MJ. Identifying randomised trials. In: Egger M, Davey Smith G, Altman DG (eds). Systematic Reviews in Health Care: Meta-analysis in Context. London: BMJ Books, 2001, pp.69–86.