Number and size of randomized trials reported in general health care journals from 1948 to 1997

Steve McDonalda,b, Margaret Westbya, Mike Clarkea and Carol Lefebvrea and The Cochrane Centres' Working Group On 50 Years Of Randomized Trials

a UK Cochrane Centre, NHS Research & Development Programme, Oxford OX2 7LG, UK.

Steve McDonald, Australasian Cochrane Centre, Monash Institute of Health Services Research, Monash Medical Centre, Locked Bag 29, Clayton, VIC 3168, Australia. E-mail: steve.mcdonald{at}med.monash.edu.au

Abstract

Background Randomized trials are important for controlling selection biases, and where sufficient numbers of participants are involved, have the potential to yield reliable estimates of treatment effects.

Methods We investigated trends in the number and size of randomized trials reported in general health care journals from 1948 to 1997. From the handsearching of 18 general health care journals we collected data on the number of reports of randomized trials in each journal per year, and the number of participants in each trial.

Results A total of 5503 reports of trials were identified in 18 general health care journals. More than a third appeared in the British Medical Journal. The peak period for trial reports was the mid 1980s, with more in 1986 than any other year (242). By the mid 1990s the number per year had declined by a third. Trials with fewer than 100 participants accounted for most of the reports (69%). In spite of the overall decline in the number of trial reports, those involving 100 participants or more continued to increase throughout the period studied.

Conclusions The continued increase in the number of larger trials reported is encouraging, especially if it represents an increase in the size of trials more generally. Further research is needed to determine whether the trends over time identified here are reflective more of trends in the actual conduct of, rather than simply the reporting, of randomized trials.

Keywords Randomized controlled trials, periodicals

Accepted 3 October 2001

In 1948 the British Medical Journal (BMJ) published the landmark trial of streptomycin for pulmonary tuberculosis.1 Although not the earliest reported randomized trial,2,3 it is credited with setting the standard for the conduct and reporting of modern randomized trials, and marks the earliest year from which members of the Cochrane Collaboration routinely search for trials.4

Efforts by the Cochrane Collaboration to identify randomized trials have resulted in over 310 000 references being made available in The Cochrane Controlled Trials Register.5,6 Cochrane centres have contributed to this activity by handsearching general health care journals, a procedure in which the complete contents of a journal are read to identify reports of trials. This study presents an analysis of the number and size of randomized trials reported in a sample of these journals.

Methods

To ensure a range of countries and languages was represented, each Cochrane centre involved in the study selected one general health care journal from one or more of the countries for which it acts as the reference centre. Where possible, this was the journal of the national medical association (or equivalent).

Journals were handsearched for the 50 years from 1948 to 1997 for all reports of trials in which participants were randomly (or quasi-randomly) assigned to alternative forms of care. Trial reports published as letters, conference abstracts and news items were excluded from this study. We analysed data on the annual number of trial reports identified in each journal, and the number of participants in each trial.

Results

Data for 18 journals were supplied by 10 Cochrane centres. Fifteen were searched for the entire period from 1948 to 1997. Seven journals were published in English, three in German, and one each in Danish, Dutch, Finnish, French, Italian, Norwegian, Spanish and Swedish (Table 1Go).


View this table:
[in this window]
[in a new window]
 
Table 1 Number of randomized trial reports in general health care journals
 
A total of 5503 trial reports were identified from a total of more than 800 journal years of searching. The largest single contribution was from the BMJ (2016 reports, 37%). The peak period for trial reports was the mid 1980s, with more in 1986 than any other year (242). By the mid 1990s the number had declined by a third to levels similar to the 1970s (Figure 1Go). This general pattern was seen in each journal, with the peak year for each mostly occurring during the 1980s.



View larger version (18K):
[in this window]
[in a new window]
 
Figure 1 Number of trial reports per year according to trial size over time

 
The number of randomized participants was extracted from 4527 (82%) of the trial reports, representing a total of more than 3.7 million people. The largest trial (1 910 093 children) was reported in the BMJ in 1957.7 To investigate trends in size, trials were grouped into those that randomized fewer than 100 participants (for ease of reference these are deemed ‘small’), and those that randomized 100 or more (‘large’ trials). Most (3139, 69%) of the trials were small.

The number of reports of large trials per year increased from a mean of 9 in the mid 1950s to 44 in the mid 1980s. In contrast to the pattern for small trials, this increased into the 1990s (to 62). The increase in the 1990s was primarily a feature of two journals (Annals of Internal Medicine and the BMJ) which account for 76% of all large trials identified between 1990–1997. The number of large trials appearing in the other journals was either declining or the trend was uncertain. The proportion of trials defined as large increased over time. During the 1950s and 1960s a third of the trials were large. This proportion fell to a quarter by the mid 1980s but then increased, so that just over half the trials had 100 or more participants by the mid 1990s.

Discussion

Our study found that, in spite of a decline in the number of trial reports appearing in general health care journals over the last 10–15 years, there was an increase throughout the 50-year period in the number of trials involving 100 or more participants.

The downturn in trial reports seen in these journals is unlikely to reflect a wider trend to conduct fewer randomized trials. A search of MEDLINE, for example, using the publication type term Randomized-Controlled-Trial reveals a large year-on-year increase since 1966 in the number of studies retrieved. In addition, a survey of 2000 controlled trials in schizophrenia,8 found largely in psychiatric journals by hand- and electronic-searching, revealed a steady increase in the annual number of trials over 50 years. More plausible explanations are the growth in the number of specialist journals, and the changing nature of general journals. A publication analysis of trials conducted by the UK Medical Research Council shows a trend towards reporting in specialist journals (I Chalmers, personal communication). For the period 1948–1967, 98% (59/60) of MRC trials were reported in the BMJ or the Lancet, but this was only 20% (32/160) for 1978–1997. Competition from specialist journals, particularly those with an international readership, is likely to be most keenly felt by the general journals with smaller circulations.

Decreases in the number of trial reports in general journals may indicate changing editorial policies and author preferences. The emergence of systematic reviews may have affected the number of trials being published in general journals, and the trend towards larger trials may suggest that editors are becoming more selective in the trials they publish. Authors themselves may be by-passing general journals, especially if the trial is small and of only limited interest, or deciding that their research will receive greater exposure in a leading specialist journal.

The trend towards an increasing number and proportion of ‘large’ trials is encouraging, although this is only apparent in the Annals of Internal Medicine and the BMJ. The number of trials involving over 1000 participants has also increased in these two journals. This might suggest that editors are becoming more selective in the trials they decide to publish, but also, hopefully, indicates a realization by researchers and funding bodies of the need to conduct trials that are of a sufficient size to detect moderate but important treatment effects.

Our study had several potential limitations. First, our sample size was small and omitted other major general journals such as JAMA and the Lancet. Second, we acknowledge that 100 is an arbitrary marker of trial size and that other factors besides absolute sample size numbers affect power calculations. Third, we did not attempt to correct for potential confounding resulting from changes in the number of pages and/or number of issues published per year in these journals. Finally, a small number of trials might have been counted in the analyses more than once, for example, where multiple reports of the same trial appeared in two or more journals.

Conclusion

Our findings have revealed a decline in the number of trial reports in general health care journals, but also a tendency for the trials that do appear to be larger. Further research is needed to determine whether these findings are reflective more of trends in the conduct of trials or in their reporting.


KEY MESSAGES

  • The 1980s was the peak period for trial reports and was followed by a downturn in the 1990s.
  • Trials with fewer than 100 participants accounted for over two-thirds of the 5500 reports.
  • In spite of the decline in the number of trial reports, the continued increase in the number of larger trials is encouraging.
  • The downturn in the trial reports seems most likely to be due to a growth in the number of specialist journals and the changing nature of general journals, rather than a reflection of fewer randomized trials being conducted.

 

Acknowledgments

Financial support: The handsearch of the European journals included in this study formed part of a European Union project under the BIOMED-1 Program [contract number BMH1-CT94-1289] which was co-ordinated at the UK Cochrane Centre. The authors thank Daniel Galandi, Kirsten Lone Jensen, Joan Marti and Jane Nguyen for contributing to the data extraction, Jim Halsey for running the statistical analyses, Iain Chalmers for permission to use unpublished material, and the many handsearchers who between them read through a combined total of over 800 years of published papers. MW is supported by a Daphne Jackson Fellowship.

Notes

b Current address: Australasian Cochrane Centre, Monash Institute of Health Services Research, Melbourne, VIC 3168, Australia. Back

Group participants: Gerd Antes (German Cochrane Centre), Xavier Bonfill (Iberoamerican Cochrane Centre), Frank Davidoff (Annals of Internal Medicine), Peter Gotzsche (Nordic Cochrane Centre), Margaret Haugh (French Cochrane Centre), Marjan Loep (Dutch Cochrane Centre), Philippa Middleton (Australasian Cochrane Centre), Elizabeth Pienaar (South African Cochrane Centre) and Vanna Pistotti (Italian Cochrane Centre).

References

1 Medical Research Council Streptomycin in Tuberculosis Trials Committee. Streptomycin treatment for pulmonary tuberculosis. Br Med J 1948;ii:769–82.

2 Royal College of Physicians of Edinburgh. Controlled trials from history. (www.rcpe.ac.uk/controlled_trials) Accessed 30 September 2001.

3 Doll R. Controlled trials: the 1948 watershed. Br Med J 1998;317: 1217–20.[Free Full Text]

4 Bero L, Rennie D. The Cochrane Collaboration. Preparing, maintaining, and disseminating systematic reviews of the effects of health care. JAMA 1995;274: 1962–64.[CrossRef][ISI][Medline]

5 Lefebvre C, Clarke MJ. Identifying randomised trials. In: Egger M, Davey Smith G, Altman DG (eds). Systematic Reviews in Health Care: Meta-analysis in Context. 2nd Edn. London: Br Med J Publishing, 2001.

6 The Cochrane Controlled Trials Register. In: The Cochrane Library. Oxford: Update Software, 2001. Updated quarterly.

7 Medical Research Council Poliomyelitis Vaccines Committee. The assessment of the British vaccine against poliomyelitis. Br Med J 1957;i:1271–77.

8 Thornley B, Adams C. Content and quality of 2000 controlled trials in schizophrenia over 50 years. Br Med J 1998;317: 1181–84.[Abstract/Free Full Text]