Effect of Early Patient Enrollment on the Time to Completion and Publication of Randomized Controlled Trials

Anna-Bettina Haidich1 and John P. A. Ioannidis1,2

1 Clinical Trials and Evidence-based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece.
2 Department of Medicine, Tufts University School of Medicine, Boston, MA.


    ABSTRACT
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
The authors evaluated whether early enrollment affects the significance of the results and the time to completion and publication of randomized controlled trials. Seventy-seven efficacy randomized controlled trials (total enrollment, 28,992 patients) initiated by the Acquired Immunodeficiency Syndrome Clinical Trials Group between 1986 and 1996 were evaluated. After adjustment for target sample size, for each 10-fold increase in the first-month accrual, the odds of a trial reaching statistically significant results increased 2.8-fold (p = 0.040). The relative enrollment during the first month over target sample size (hazard ratio (HR) = 1.40 per 10 percent increase, p = 0.004) and masking (HR = 1.78 for double-blind vs. single or unblinded studies, p = 0.031) were the major predictors of faster completion. Rapid early accrual (HR = 1.09 per 10 additional patients accrued the first month, p = 0.011) and statistical significance in favor of an experimental arm (HR = 2.47, p = 0.004) independently predicted faster publication. Early enrollment is a strong predictor of whether a study will reach formal statistical significance, and it can offer predictive information on the time needed to complete the study and publish its findings. Ongoing unpublished studies and their enrollment rates may need to be considered when interpreting the accumulated evidence.

bias (epidemiology); double-blind method; patient participation; patients; randomized controlled trials; sample size; statistics

Abbreviations: ACTG, Acquired Immunodeficiency Syndrome Clinical Trials Group; AIDS, acquired immunodeficiency syndrome; AUC, area under the curve; CI, confidence interval; HIV, human immunodeficiency virus; HR, hazard ratio; RCT, randomized controlled trial; ROC, receiver operating characteristic


    INTRODUCTION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Efficacy randomized controlled trials (RCTs) have a pivotal role in the adoption of preventive and therapeutic interventions in clinical practice. These experiments may take considerable time to conduct and publish, since they require the recruitment of an adequate number of subjects and may also need substantial follow-up of these subjects to assess outcomes. Then additional time is spent to analyze the results, prepare the respective manuscript, and undergo peer review and publication. Empirical data suggest that the median time from initiation of an efficacy trial to the publication of its results is more than 5 years (1Go). However, there is substantial variability: Some trials take about 2 years to run their cycle from design to dissemination of results, while others remain unpublished 10 years or more after their initiation. Some trials accomplish their aims, while others fail and are abandoned, and still others are protracted. However, what parameters determine this variable fate of these clinical experiments?

Previous work has shown that randomized efficacy trials are published more quickly when they find formally statistically significant results (1GoGoGoGoGoGoGoGo–9Go), while "negative" studies may remain unpublished for much longer periods of time. However, the statistical significance of the results is known only once the trial is completed and the data are analyzed. It is unknown whether we could predict the fate of a RCT on the basis of its study characteristics and early information about its conduct. In this regard, it would be interesting to investigate whether the early ability of a trial to recruit subjects in the first few months after its initiation can offer some insight about its long-term fate. There is evidence that the pace of early enrollment in the first 2 months may be related to the eventual ability of a trial to attain its target sample size (10Go). The recruitment of patients is routinely recorded in all randomized trials as an indicator of a trial's progress over time. Important questions arise: Can we predict whether a trial will be quickly completed and published based on its early accrual? Furthermore, are there early signs that a trial is unlikely to materialize and may even remain an unpublished experiment? To address these issues, we performed an empirical assessment using a large prospective registry of randomized efficacy trials launched by a multicenter clinical trials group over a period of 10 years.


    MATERIALS AND METHODS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Database
We used data on the accrual of patients in RCTs that were initiated by the Acquired Immunodeficiency Syndrome Clinical Trials Group (ACTG) between October 10, 1986 and October 10, 1996. All patients enrolled until November 12, 1999, were considered. ACTG is sponsored by the National Institute of Allergy and Infectious Diseases, and it represents the largest network for the conduct of clinical trials on human immunodeficiency virus (HIV) infection and its complications worldwide. ACTG performs trials both in adults (adult ACTG) and in children (pediatric ACTG). It uses the clinical resources of 30 university sites across the United States as well as other collaborative clinical units. Sample size estimates for "target" enrollment are routinely obtained for randomized efficacy trials before a trial is launched.

Study selection
As in previous work (1Go, 10Go), we considered only the randomized controlled efficacy trials, which had been designated as phase II, II/III, or III by their investigators. Observational, nonrandomized, pharmacokinetic, phase I, and phase I/II studies were excluded as well as substudies of the main protocols. Qualification for inclusion was based on examination of the complete protocols (1Go). Trials were selected regardless of whether they compared a regimen with placebo, different regimens, or doses of the same medication. All protocols that have enrolled any patients have been registered in a prospective registry of ACTG trials maintained by the Division of Acquired Immunodeficiency Syndrome (AIDS) at the National Institutes of Health.

For this analysis, studies that were also jointly funded by other organizations (such as pharmaceutical companies, the Terry Beirn Community Programs for Clinical Research on AIDS, or the Studies of Ocular Complications in AIDS research group) were excluded whenever only data from ACTG-funded patients were available and the ACTG-funded patients alone accounted for less than 80 percent of the target enrollment.

Data and variables considered
Trial characteristics included the actual sample size, the population (adult or pediatric), the trial domain (antiretroviral therapy or complications of HIV, including opportunistic infections and neurologic complications), the masking (double-blind vs. single-blind or unmasked), and the place where data were managed (pharmaceutical industry or other).

On-study dates were used to calculate the number of patients enrolled over time for each trial. The date of starting enrollment for each trial was defined as the date the first patient entered the study in any of the participating sites. In multicenter studies, recruitment unavoidably starts at different time points at different sites. Early accrual metrics reflect both the efficiency of recruiting patients once sites are open and the efficiency of sites in avoiding potential delays related to the timing of the ethical review and other local parameters. In ACTG, ethical review is typically completed efficiently and sites open at about the same time. Exceptions may occur, and such exceptions may be more prominent in other settings.

We considered the following parameters that characterize early enrollment (early enrollment metrics): 1) the number of patients accrued during the first month; 2) the number of patients accrued during the first 2 months; 3) the ratio of patients accrued during the first month over the target sample size; 4) the ratio of patients accrued during the first 2 months over the target sample size; and 5) the ratio of patients accrued during the first 3 months over the target sample size.

For all studies, we recorded the date of completion of follow-up and of final publication and the level of statistical significance of the analysis of their main outcome. For studies that continued follow-up beyond their primary analysis and publication, follow-up was censored at the time of the primary analysis. All data were censored on November 12, 1999. Trials with nonstatistically significant findings (p >= 0.05) or formally favoring the control arm (p < 0.05) are called "negative." Trials formally favoring an experimental arm (p < 0.05) are categorized as "positive" (1Go).

Statistical analysis
First, we investigated whether trials with more rapid early enrollment were more likely to reach 1) statistically significant results in favor of any arm or 2) positive results, as defined above. Each early enrollment metric was fit in a univariate logistic regression (11Go) against each of these two outcomes. Logarithmic transformations of enrollment metrics were used if they had a better fit than the absolute values. We also performed logistic regressions, adjusting for the target sample size. Additionally, we compared the mean proportion of target finally achieved (accrual/target) in trials with "significant" versus "nonsignificant" results and in trials with positive versus negative results, using the Mann-Whitney U test.

Second, using Cox models, we evaluated whether early enrollment metrics were predictive of the time from start to completion of follow-up, completion of follow-up to publication, and start of enrollment to publication. Quartiles of each early enrollment metric were plotted separately with Kaplan-Meier plots to confirm that there was no obvious violation of proportional hazards (12Go). Adjusted multivariate Cox models also considered other trial characteristics that were found to be significant predictors of the time to completion and the time to publication in univariate analyses. The final models were built with forward selection of variables according to likelihood ratio criteria. Multivariate models were shown graphically with Kaplan-Meier plots considering combinations of the independent predictors.

Period effects were considered by evaluating whether the calendar year of start or completion was a significant predictor of the time to completion or publication or of the time from completion to publication, but no significant associations were found (data not shown).

To illustrate the predictive ability of early enrollment metrics, we also performed receiver operating characteristic (ROC) curve analyses using the following outcomes: publication in fewer than 4 years from starting enrollment and completion in fewer than 2 years from starting enrollment. The area under the curve (AUC) of the ROC curves was estimated. Representative pairs of sensitivity and specificity for various values of early accrual metrics are reported.

For the analyses of time to completion and publication, we also excluded studies considered to be early protocol failures, that is, studies abandoned early by their investigators due to futility because fewer than 20 patients (typically fewer than six) had been enrolled after a few months of enrollment. These trials are excluded because abandonment is not equivalent to completion, and moreover, these studies are unlikely ever to be published.

Statistical analyses were conducted in SPSS 10.0 (SPSS, Inc., Chicago, Illinois). All p values are two-tailed.


    RESULTS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Characteristics of registered trials
A total of 77 randomized efficacy trials with total enrollment of 28,992 patients were considered. Of these, seven were closed early, having failed to accrue more than 20 patients (total enrollment = 49 patients in all protocol failures). Of the remaining 70 studies, one was still open to accrual, one was closed to accrual and continuing follow-up, and 68 (with 28,443 patients) had been completed; 45 completed trials had been published at the time data were censored for analysis (table 1).


View this table:
[in this window]
[in a new window]
 
TABLE 1. Characteristics of trials, AIDS Clinical Trials Group

 
Determinants of statistically significant or positive trial results
The absolute early accrual and, to a lesser extent, metrics expressing the proportion of the target enrollment achieved in the first months were positively associated with finding statistically significant results in the final analysis of a trial. The results were similar when positive trials were considered (table 2). The models were further adjusted for the target sample size, which also predicted whether a trial reached statistical significance. The adjusted effects of the early enrollment metrics were qualitatively unchanged. For example, adjusting for target sample size, for each 10-fold increase in the first-month accrual or in the first 2 months' accrual, the odds of the trial reaching statistically significant results increased 2.8-fold (p = 0.040 or p = 0.048, respectively). The results were similar when the protocol failures were excluded (not shown).


View this table:
[in this window]
[in a new window]
 
TABLE 2. Logistic regressions for prediction of statistically significant trial results or "positive" trials, AIDS Clinical Trials Group

 
There was no significant difference in the mean proportion of target finally achieved in trials with significant results and those with nonsignificant results or in trials with positive and those with negative results (p = 0.24 and p = 0.30, respectively).

Predictors of time to completion and publication
Early accrual in the first 1 or 2 months was a major predictor of the time from starting enrollment to the publication of a clinical trial (table 3). For example, the rate of publication increased 1.12-fold for every 10 additional patients enrolled during the first month. The absolute early enrollment was also predictive of the time from completion to publication. For example, the rate of publication after completion increased 1.09-fold for every 10 additional patients enrolled in the first month. The relative accrual in the first 1–3 months over the target sample size was not as strongly associated with the time to publication (table 3).


View this table:
[in this window]
[in a new window]
 
TABLE 3. Cox regressions for prediction of time to study completion and publication, AIDS Clinical Trials Group

 
Early enrollment was also predictive of the time it took to complete a trial. Both the absolute early enrollment and the relative enrollment over the target sample size were significant predictors of earlier completion (table 3). The rate of completion was 1.07 times faster for every 10 additional patients accrued during the first month (95 percent confidence interval (CI): 1.01, 1.13) or 1.03 times faster for every 10 additional patients accrued during the first 2 months (95 percent CI: 1.00, 1.06).

Other predictors and adjusted analyses
In univariate analyses, the time from start to publication was also shorter for positive trials (hazard ratio (HR) = 2.8, 95 percent CI: 1.5, 5.1), for statistically significant trial results (HR = 2.7, 95 percent CI: 1.5, 5.0), and for larger trials (HR = 2.4, 95 percent CI: 1.2, 4.8 for every 10-fold increase in target sample size). The same parameters also affected the time from completion to publication (HR = 2.8, 95 percent CI: 1.5, 5.1 for positive trials; HR = 2.5, 95 percent CI: 1.3, 4.6 for statistically significant trial results; HR = 2.9, 95 percent CI: 1.4, 5.8 for every 10-fold increase in target sample size; and HR = 2.5, 95 percent CI: 1.3, 4.7 for every 10-fold increase in achieved sample size, respectively). In multivariate modeling (table 4), the first-month accrual and the presence of positive findings were the only significant independent predictors of the time from start to publication. Thus, the rate of publication increased 1.09-fold for every 10 additional patients accrued during the first month, after adjustment for whether the trial was positive or negative. Positive findings and a larger achieved sample size were the key independent determinants of a shorter time to publication after completion (table 4).


View this table:
[in this window]
[in a new window]
 
TABLE 4. Multivariate Cox regression models for the prediction of time to study completion and publication, AIDS Clinical Trials Group

 
In univariate regressions, the time to completion was shorter in double-blind trials (HR = 1.6, 95 percent CI: 1.0, 2.7), trials with data management performed by the pharmaceutical industry (HR = 3.3, 95 percent CI: 1.0, 10.8), antiretroviral studies (HR = 1.6, 95 percent CI: 1.0, 2.6), and studies with statistically significant results (HR = 1.6, 95 percent CI: 1.0, 2.7). In multivariate modeling, the ratio of first-month accrual over target along with the masking were the only significant independent determinants of the time from start to completion (table 4). The rate of completion was 40 percent faster for every 10 percent increase in the relative enrollment over the target sample size during the first month, after adjustment for whether or not the trial was double-blind.

The magnitude of the effect of early accrual on the time to completion and publication is shown in figure 1, adjusting for other important parameters. The median time from start of enrollment to publication was 3.9 years for positive trials with greater than or equal to eight patients accrued in the first month (above median first-month enrollment) versus 6.5 for negative trials with lower (below median) first-month enrollment (figure 1). The median time from start to completion of follow-up was 2.0 years for double-blind studies in which greater than or equal to 4 percent of the target sample size was enrolled in the first month (above median relative first-month enrollment) versus 3.8 years for single-blind or unmasked studies with lower (below median) relative first-month enrollment (figure 2).



View larger version (12K):
[in this window]
[in a new window]
 
FIGURE 1. Time to publication from start of enrollment for positive or negative trials with high (above median) or low (below median) first-month enrollment. Protocol failures and trials with pending results are excluded. Log rank adjusted for trend, p = 0.002. AIDS Clinical Trials Group.

 


View larger version (12K):
[in this window]
[in a new window]
 
FIGURE 2. Time to completion of follow-up from start of enrollment for double-blind (DB) or single-blind and unmasked studies (not DB) with high (above median) or low (below median) first-month enrollment over target sample size. Protocol failures are excluded. Log rank adjusted for trend, p = 0.001. AIDS Clinical Trials Group.

 
ROC analysis
Early accrual metrics had modest predictive ability for determining whether a trial would be completed in fewer than 2 years and whether it would be published in fewer than 4 years. For example, the first month's accrual had an AUC of 0.68 for publication in fewer than 4 years (AUC = 0.66, excluding failed protocols). Accrual of less than eight patients in the first month (the median) had a sensitivity of 69 percent and a specificity of 56 percent for predicting lack of publication within less than 4 years (excluding failed protocols). The ratio of the first-month accrual over the target sample size had an AUC of 0.72 for completion of a trial in fewer than 2 years, excluding the failed protocols (AUC = 0.73 considering failed protocols as uncompleted). Accrual of less than 3.8 percent of the target in the first month (the median) had a sensitivity of 75 percent and specificity of 60 percent for predicting lack of completion within 2 years. The respective numbers were 80 and 60 percent when failed protocols were considered as uncompleted.


    DISCUSSION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
This analysis shows that early accrual during the first months of a randomized efficacy trial may predict whether statistically significant or positive results are eventually found for the main outcome in the final analysis. We have shown previously (10Go) that the more patients enrolled during the first months the more likely were the RCTs to reach their target sample size. Attaining the target sample size also implies no loss of power to detect significant differences between the study arms in the final analysis. The association between early accrual and eventual statistical significance was not attributed to the planned magnitude of the trial. After adjustment for the target sample size, the early enrollment metrics were still predictive of whether a study would reach formal statistical significance or not.

Early accrual may be determined by several study-specific factors and may also reflect the adequacy of the network of participating clinical sites, the quality of the study design, the attractiveness of the trial design, and the tested treatment to patients who are candidates for recruitment. Of course, a trial reaching a nonstatistically significant result is not inferior to a trial reaching formally statistically significant conclusions (13Go), especially when both trials had been appropriately designed to have comparable power and were able to reach their target sample size. Otherwise, a nonsignificant result may simply reflect the fact that the trial was poorly designed or failed to reach its aims. Interestingly, in this large sample of studies, trials with significant results reached, on average, as close to their target sample size as those with nonsignificant results. Nevertheless, there may be a greater demand for patient participation in the pivotal trials of new treatments that are eventually shown to be comparatively more effective than the standard available treatment(s). In a field such as HIV infection, a large patient pool that has been failing standard treatment and is eager to try promising new regimens has often existed. Early trials with surrogate markers may provide indirect evidence about the eventual clinical efficacy of a new regimen. Thus, patient enrollment may be more enthusiastic in studies evaluating drugs eventually proven to be effective. The early dynamics of patient recruitment may be a signal about the efficacy or lack thereof of the tested treatments.

Early accrual in the first 1 or 2 months was a major predictor of the time from start of enrollment to completion of follow-up. Enrollment during the first months may help to estimate how long it will take to complete a study. This would be expected, especially if we assume that the rate of accrual is uniform over time, and our data further confirm that this tends to be the case. Therefore, such information would be helpful in steering a trial. Masking was also an independent predictor of the time from start to completion. On average, open-label trials may be designed with a longer anticipated follow-up than double-blind studies. Alternatively, perhaps double-blinding allows better control of bias (14GoGo–16Go) and may be a quality characteristic that correlates with the ability of a trial to accomplish its goals earlier. The median time from start of enrollment until completion of follow-up was almost half for double-blind studies with high (above-median) early relative accrual compared with single-blind or unblinded studies with low (below-median) early relative accrual (2.0 vs. 3.8 years).

Enrollment during the first months was also strongly related to the time from start of enrollment to publication and the time from completion of follow-up to publication. As we have shown previously (1Go), positive results are the strongest predictor of rapid publication after completion of follow-up, and large trials, once completed, are also more rapidly published than small ones. These two parameters seemed more important than the early accrual in determining the fate of a trial after its completion. Our findings thus provide further evidence for the presence of "publication bias" or, more appropriately, "time lag bias" (1Go) for negative findings originating from relatively small trials. This bias occurs when, among two equally well designed and informative trials, publication of the trial with statistically nonsignificant results is delayed. However, when we considered the total time it took for a trial to materialize and be disseminated, early accrual offered independent information beyond the statistical significance of the results, and it was more important than the trial sample size in determining the total time from start of enrollment to publication.

Although early enrollment metrics can offer predictive information on the time needed to complete the study and publish its findings, we should caution that the strong statistical associations that we observed translate to modest AUC values and that misclassification is not uncommon. Nevertheless, a slow starter trial is at considerable disadvantage for reaching its aims.

The time lag of trials with different enrollment patterns and different levels of statistical significance may have implications for meta-analyses and for assessment of the total randomized evidence in various fields (17Go). Slow-enrolling studies and studies with negative results may appear later than rapidly completed trials with more "impressive" findings, and they may change our belief about the apparent efficacy of various treatments (18Go). Thus, in conducting meta-analyses, it would be important to examine whether there are still "pending" ongoing studies in the field, as well as to know the pace of their progress. The results of early-appearing studies and the conclusions of early meta-analyses may sometimes be more optimistic than the final picture that emerges when all pieces of the randomized evidence become available (19Go).

We evaluated a highly structured, multicenter network with standing committees and considerable infrastructure support. In other multicenter trials, clinical sites might not enter in the same pace, and the early enrollment might be slower. For example, staggered ethical approval of clinical sites may be more common in other settings. Furthermore, in some trials, additional sites may be recruited even during the conduct of the study, while ACTG typically uses a fixed number of participating sites and sites that join later are not frequent. Even with these limitations, early accrual metrics also reflect the efficiency of the organizational mechanics behind a clinical trial team. Trials with poor organization that are inefficient in recruiting, approving, and opening sites may have both slow early enrollment and delayed completion and publication. To evaluate the generalizability of our results, it might be useful to study additional trial groups in the future. Nevertheless, the ACTG represents the largest multicenter clinical trials group in the HIV field and one of the largest irrespective of discipline, and thus, it may be difficult to assemble a similar amount of data in other fields.

Our study was limited to trials from the field of HIV infection. Perhaps trials with relatively slow early enrollment may still be able to materialize and be disseminated promptly in other fields in which there is less time pressure and in which the therapeutic background is less likely to change rapidly during the trial conduct. However, randomized trials are costly experiments (20Go), and a slow protracted enrollment is likely to be a nuisance in any field. Moreover, in HIV infection, changes in the course of the AIDS epidemic might affect the available patient pools over time and also affect their heterogeneity (21Go, 22Go). Enrollment may thus be more foreseeable, and its effects on the fate of a trial may be even more predictable in other areas of research in which the prevalent patient pools are steadies and changes in therapeutics are less dramatic.


    ACKNOWLEDGMENTS
 
Anna-Bettina Haidich was supported in part by the General Secretariat for Research and Technology, Greece, through the European Union (grant PENED 99ED27–974).

The authors are grateful to Annice Bergeris from the Division of AIDS, National Institute of Allergy and Infectious Diseases, National Institutes of Health, for input in data management and to Dr. Robert Schooley from the University of Colorado Health Sciences Center and the AIDS Clinical Trials Research Group leadership for allowing the use of their database and for helpful comments.


    NOTES
 
Correspondence to Dr. John P. A. Ioannidis, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece (e-mail: jioannid{at}cc.uoi.gr).


    REFERENCES
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281–6.[Abstract/Free Full Text]
  2. Easterbrook PJ, Berlin JA, Gopalan R, et al. Publication bias in clinical research. Lancet 1991;337:867–72.[ISI][Medline]
  3. Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;DOC 50:4967.
  4. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;263:1385–9.[Abstract]
  5. Dickersin K, Min YI. Publication bias: the problem that won't go away. Ann N Y Acad Sci 1993;703;135–48.
  6. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267:374–8.[Abstract]
  7. Hetherington J, Dickersin K, Chalmers I, et al. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989;84:374–80.[Abstract]
  8. Schrerer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts: a meta-analysis. JAMA 1994;272:158–62.[Abstract]
  9. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640–5.[Abstract/Free Full Text]
  10. Haidich A-B, Ioannidis JPA. Patterns of patient enrollment in randomized controlled trials. J Clin Epidemiol 2001;54:877–83.[ISI][Medline]
  11. Agresti A. An introduction to categorical data analysis. New York, NY: John Wiley & Sons, 1996.
  12. Collett D. Modelling survival data in medical research. London, England: Chapman & Hall, 1994.
  13. Angell M. Negative studies. N Engl J Med 1989;321:464–6.[ISI][Medline]
  14. Piantadosi S. Clinical trials: a methodologic perspective. New York, NY: John Wiley & Sons, 1997.
  15. Day SJ, Altman DG. Statistics notes: blinding in clinical trials and other studies. (Review). BMJ 2000;321:504.[Free Full Text]
  16. Schulz KF, Chalmers I, Hayes RJ, et al. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–12.[Abstract]
  17. Clarke MJ, Stewart LA. Time lag bias in publishing clinical trials. (Letter). JAMA 1998;279:1952.[Free Full Text]
  18. Ioannidis JPA, Lau J. Evolution of treatment effects over time: empirical insight from recursive cumulative meta-analyses. Proc Natl Acad Sci U S A 2001;98:831–6.[Abstract/Free Full Text]
  19. Ioannidis JPA, Contopoulos-Ioannidis DG, Lau J. Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. J Clin Epidemiol 1999;52:281–91.[ISI][Medline]
  20. Detsky AS. Using cost-effectiveness analysis to improve the efficiency of allocating funds to clinical trials. Stat Med 1990;9:173–84.[ISI][Medline]
  21. Ioannidis JP, Cappelleri JC, Schmid CH, et al. Impact of epidemic and individual heterogeneity on the population distribution of disease progression rates: an example from patient populations in trials of human immunodeficiency virus infection. Am J Epidemiol 1996;144:1074–85.[Abstract]
  22. Ioannidis JP, Lau J. Heterogeneity of the baseline risk within patient populations of clinical trials: a proposed evaluation algorithm. Am J Epidemiol 1998;148:1117–26.[Abstract]
Received for publication January 5, 2001. Accepted for publication June 26, 2001.