1 Clinical Trials and Evidence-based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece.
2 Department of Medicine, Tufts University School of Medicine, Boston, MA.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
bias (epidemiology); double-blind method; patient participation; patients; randomized controlled trials; sample size; statistics
Abbreviations: ACTG, Acquired Immunodeficiency Syndrome Clinical Trials Group; AIDS, acquired immunodeficiency syndrome; AUC, area under the curve; CI, confidence interval; HIV, human immunodeficiency virus; HR, hazard ratio; RCT, randomized controlled trial; ROC, receiver operating characteristic
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Previous work has shown that randomized efficacy trials are published more quickly when they find formally statistically significant results (19
), while "negative" studies may remain unpublished for much longer periods of time. However, the statistical significance of the results is known only once the trial is completed and the data are analyzed. It is unknown whether we could predict the fate of a RCT on the basis of its study characteristics and early information about its conduct. In this regard, it would be interesting to investigate whether the early ability of a trial to recruit subjects in the first few months after its initiation can offer some insight about its long-term fate. There is evidence that the pace of early enrollment in the first 2 months may be related to the eventual ability of a trial to attain its target sample size (10
). The recruitment of patients is routinely recorded in all randomized trials as an indicator of a trial's progress over time. Important questions arise: Can we predict whether a trial will be quickly completed and published based on its early accrual? Furthermore, are there early signs that a trial is unlikely to materialize and may even remain an unpublished experiment? To address these issues, we performed an empirical assessment using a large prospective registry of randomized efficacy trials launched by a multicenter clinical trials group over a period of 10 years.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Study selection
As in previous work (1, 10
), we considered only the randomized controlled efficacy trials, which had been designated as phase II, II/III, or III by their investigators. Observational, nonrandomized, pharmacokinetic, phase I, and phase I/II studies were excluded as well as substudies of the main protocols. Qualification for inclusion was based on examination of the complete protocols (1
). Trials were selected regardless of whether they compared a regimen with placebo, different regimens, or doses of the same medication. All protocols that have enrolled any patients have been registered in a prospective registry of ACTG trials maintained by the Division of Acquired Immunodeficiency Syndrome (AIDS) at the National Institutes of Health.
For this analysis, studies that were also jointly funded by other organizations (such as pharmaceutical companies, the Terry Beirn Community Programs for Clinical Research on AIDS, or the Studies of Ocular Complications in AIDS research group) were excluded whenever only data from ACTG-funded patients were available and the ACTG-funded patients alone accounted for less than 80 percent of the target enrollment.
Data and variables considered
Trial characteristics included the actual sample size, the population (adult or pediatric), the trial domain (antiretroviral therapy or complications of HIV, including opportunistic infections and neurologic complications), the masking (double-blind vs. single-blind or unmasked), and the place where data were managed (pharmaceutical industry or other).
On-study dates were used to calculate the number of patients enrolled over time for each trial. The date of starting enrollment for each trial was defined as the date the first patient entered the study in any of the participating sites. In multicenter studies, recruitment unavoidably starts at different time points at different sites. Early accrual metrics reflect both the efficiency of recruiting patients once sites are open and the efficiency of sites in avoiding potential delays related to the timing of the ethical review and other local parameters. In ACTG, ethical review is typically completed efficiently and sites open at about the same time. Exceptions may occur, and such exceptions may be more prominent in other settings.
We considered the following parameters that characterize early enrollment (early enrollment metrics): 1) the number of patients accrued during the first month; 2) the number of patients accrued during the first 2 months; 3) the ratio of patients accrued during the first month over the target sample size; 4) the ratio of patients accrued during the first 2 months over the target sample size; and 5) the ratio of patients accrued during the first 3 months over the target sample size.
For all studies, we recorded the date of completion of follow-up and of final publication and the level of statistical significance of the analysis of their main outcome. For studies that continued follow-up beyond their primary analysis and publication, follow-up was censored at the time of the primary analysis. All data were censored on November 12, 1999. Trials with nonstatistically significant findings (p 0.05) or formally favoring the control arm (p < 0.05) are called "negative." Trials formally favoring an experimental arm (p < 0.05) are categorized as "positive" (1
).
Statistical analysis
First, we investigated whether trials with more rapid early enrollment were more likely to reach 1) statistically significant results in favor of any arm or 2) positive results, as defined above. Each early enrollment metric was fit in a univariate logistic regression (11) against each of these two outcomes. Logarithmic transformations of enrollment metrics were used if they had a better fit than the absolute values. We also performed logistic regressions, adjusting for the target sample size. Additionally, we compared the mean proportion of target finally achieved (accrual/target) in trials with "significant" versus "nonsignificant" results and in trials with positive versus negative results, using the Mann-Whitney U test.
Second, using Cox models, we evaluated whether early enrollment metrics were predictive of the time from start to completion of follow-up, completion of follow-up to publication, and start of enrollment to publication. Quartiles of each early enrollment metric were plotted separately with Kaplan-Meier plots to confirm that there was no obvious violation of proportional hazards (12). Adjusted multivariate Cox models also considered other trial characteristics that were found to be significant predictors of the time to completion and the time to publication in univariate analyses. The final models were built with forward selection of variables according to likelihood ratio criteria. Multivariate models were shown graphically with Kaplan-Meier plots considering combinations of the independent predictors.
Period effects were considered by evaluating whether the calendar year of start or completion was a significant predictor of the time to completion or publication or of the time from completion to publication, but no significant associations were found (data not shown).
To illustrate the predictive ability of early enrollment metrics, we also performed receiver operating characteristic (ROC) curve analyses using the following outcomes: publication in fewer than 4 years from starting enrollment and completion in fewer than 2 years from starting enrollment. The area under the curve (AUC) of the ROC curves was estimated. Representative pairs of sensitivity and specificity for various values of early accrual metrics are reported.
For the analyses of time to completion and publication, we also excluded studies considered to be early protocol failures, that is, studies abandoned early by their investigators due to futility because fewer than 20 patients (typically fewer than six) had been enrolled after a few months of enrollment. These trials are excluded because abandonment is not equivalent to completion, and moreover, these studies are unlikely ever to be published.
Statistical analyses were conducted in SPSS 10.0 (SPSS, Inc., Chicago, Illinois). All p values are two-tailed.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
|
Predictors of time to completion and publication
Early accrual in the first 1 or 2 months was a major predictor of the time from starting enrollment to the publication of a clinical trial (table 3). For example, the rate of publication increased 1.12-fold for every 10 additional patients enrolled during the first month. The absolute early enrollment was also predictive of the time from completion to publication. For example, the rate of publication after completion increased 1.09-fold for every 10 additional patients enrolled in the first month. The relative accrual in the first 13 months over the target sample size was not as strongly associated with the time to publication (table 3).
|
Other predictors and adjusted analyses
In univariate analyses, the time from start to publication was also shorter for positive trials (hazard ratio (HR) = 2.8, 95 percent CI: 1.5, 5.1), for statistically significant trial results (HR = 2.7, 95 percent CI: 1.5, 5.0), and for larger trials (HR = 2.4, 95 percent CI: 1.2, 4.8 for every 10-fold increase in target sample size). The same parameters also affected the time from completion to publication (HR = 2.8, 95 percent CI: 1.5, 5.1 for positive trials; HR = 2.5, 95 percent CI: 1.3, 4.6 for statistically significant trial results; HR = 2.9, 95 percent CI: 1.4, 5.8 for every 10-fold increase in target sample size; and HR = 2.5, 95 percent CI: 1.3, 4.7 for every 10-fold increase in achieved sample size, respectively). In multivariate modeling (table 4), the first-month accrual and the presence of positive findings were the only significant independent predictors of the time from start to publication. Thus, the rate of publication increased 1.09-fold for every 10 additional patients accrued during the first month, after adjustment for whether the trial was positive or negative. Positive findings and a larger achieved sample size were the key independent determinants of a shorter time to publication after completion (table 4).
|
The magnitude of the effect of early accrual on the time to completion and publication is shown in figure 1, adjusting for other important parameters. The median time from start of enrollment to publication was 3.9 years for positive trials with greater than or equal to eight patients accrued in the first month (above median first-month enrollment) versus 6.5 for negative trials with lower (below median) first-month enrollment (figure 1). The median time from start to completion of follow-up was 2.0 years for double-blind studies in which greater than or equal to 4 percent of the target sample size was enrolled in the first month (above median relative first-month enrollment) versus 3.8 years for single-blind or unmasked studies with lower (below median) relative first-month enrollment (figure 2).
|
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Early accrual may be determined by several study-specific factors and may also reflect the adequacy of the network of participating clinical sites, the quality of the study design, the attractiveness of the trial design, and the tested treatment to patients who are candidates for recruitment. Of course, a trial reaching a nonstatistically significant result is not inferior to a trial reaching formally statistically significant conclusions (13), especially when both trials had been appropriately designed to have comparable power and were able to reach their target sample size. Otherwise, a nonsignificant result may simply reflect the fact that the trial was poorly designed or failed to reach its aims. Interestingly, in this large sample of studies, trials with significant results reached, on average, as close to their target sample size as those with nonsignificant results. Nevertheless, there may be a greater demand for patient participation in the pivotal trials of new treatments that are eventually shown to be comparatively more effective than the standard available treatment(s). In a field such as HIV infection, a large patient pool that has been failing standard treatment and is eager to try promising new regimens has often existed. Early trials with surrogate markers may provide indirect evidence about the eventual clinical efficacy of a new regimen. Thus, patient enrollment may be more enthusiastic in studies evaluating drugs eventually proven to be effective. The early dynamics of patient recruitment may be a signal about the efficacy or lack thereof of the tested treatments.
Early accrual in the first 1 or 2 months was a major predictor of the time from start of enrollment to completion of follow-up. Enrollment during the first months may help to estimate how long it will take to complete a study. This would be expected, especially if we assume that the rate of accrual is uniform over time, and our data further confirm that this tends to be the case. Therefore, such information would be helpful in steering a trial. Masking was also an independent predictor of the time from start to completion. On average, open-label trials may be designed with a longer anticipated follow-up than double-blind studies. Alternatively, perhaps double-blinding allows better control of bias (1416
) and may be a quality characteristic that correlates with the ability of a trial to accomplish its goals earlier. The median time from start of enrollment until completion of follow-up was almost half for double-blind studies with high (above-median) early relative accrual compared with single-blind or unblinded studies with low (below-median) early relative accrual (2.0 vs. 3.8 years).
Enrollment during the first months was also strongly related to the time from start of enrollment to publication and the time from completion of follow-up to publication. As we have shown previously (1), positive results are the strongest predictor of rapid publication after completion of follow-up, and large trials, once completed, are also more rapidly published than small ones. These two parameters seemed more important than the early accrual in determining the fate of a trial after its completion. Our findings thus provide further evidence for the presence of "publication bias" or, more appropriately, "time lag bias" (1
) for negative findings originating from relatively small trials. This bias occurs when, among two equally well designed and informative trials, publication of the trial with statistically nonsignificant results is delayed. However, when we considered the total time it took for a trial to materialize and be disseminated, early accrual offered independent information beyond the statistical significance of the results, and it was more important than the trial sample size in determining the total time from start of enrollment to publication.
Although early enrollment metrics can offer predictive information on the time needed to complete the study and publish its findings, we should caution that the strong statistical associations that we observed translate to modest AUC values and that misclassification is not uncommon. Nevertheless, a slow starter trial is at considerable disadvantage for reaching its aims.
The time lag of trials with different enrollment patterns and different levels of statistical significance may have implications for meta-analyses and for assessment of the total randomized evidence in various fields (17). Slow-enrolling studies and studies with negative results may appear later than rapidly completed trials with more "impressive" findings, and they may change our belief about the apparent efficacy of various treatments (18
). Thus, in conducting meta-analyses, it would be important to examine whether there are still "pending" ongoing studies in the field, as well as to know the pace of their progress. The results of early-appearing studies and the conclusions of early meta-analyses may sometimes be more optimistic than the final picture that emerges when all pieces of the randomized evidence become available (19
).
We evaluated a highly structured, multicenter network with standing committees and considerable infrastructure support. In other multicenter trials, clinical sites might not enter in the same pace, and the early enrollment might be slower. For example, staggered ethical approval of clinical sites may be more common in other settings. Furthermore, in some trials, additional sites may be recruited even during the conduct of the study, while ACTG typically uses a fixed number of participating sites and sites that join later are not frequent. Even with these limitations, early accrual metrics also reflect the efficiency of the organizational mechanics behind a clinical trial team. Trials with poor organization that are inefficient in recruiting, approving, and opening sites may have both slow early enrollment and delayed completion and publication. To evaluate the generalizability of our results, it might be useful to study additional trial groups in the future. Nevertheless, the ACTG represents the largest multicenter clinical trials group in the HIV field and one of the largest irrespective of discipline, and thus, it may be difficult to assemble a similar amount of data in other fields.
Our study was limited to trials from the field of HIV infection. Perhaps trials with relatively slow early enrollment may still be able to materialize and be disseminated promptly in other fields in which there is less time pressure and in which the therapeutic background is less likely to change rapidly during the trial conduct. However, randomized trials are costly experiments (20), and a slow protracted enrollment is likely to be a nuisance in any field. Moreover, in HIV infection, changes in the course of the AIDS epidemic might affect the available patient pools over time and also affect their heterogeneity (21
, 22
). Enrollment may thus be more foreseeable, and its effects on the fate of a trial may be even more predictable in other areas of research in which the prevalent patient pools are steadies and changes in therapeutics are less dramatic.
![]() |
ACKNOWLEDGMENTS |
---|
The authors are grateful to Annice Bergeris from the Division of AIDS, National Institute of Allergy and Infectious Diseases, National Institutes of Health, for input in data management and to Dr. Robert Schooley from the University of Colorado Health Sciences Center and the AIDS Clinical Trials Research Group leadership for allowing the use of their database and for helpful comments.
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|