From the Department of Biostatistics, Rollins School of Public Health, Emory University, 1518 Clifton Road NE, Atlanta, GA 30322.
![]() |
ABSTRACT |
---|
bias (epidemiology); epidemiologic methods; HIV; influenza; vaccines; validation sets
Abbreviations: IP, incidence proportion; VE, vaccine efficacy
![]() |
INTRODUCTION |
---|
However, samples or cultures are often expensive or difficult to collect, so a less specific case definition is used. In an influenza study, a nonspecific case definition might be "any respiratory illness" (3) or "febrile upper respiratory tract illness" (4
). Thus, many ascertained cases are not cases of the disease for which vaccination confers protection. This severely attenuates efficacy estimates. For instance, using only culture-confirmed cases, Belshe et al. (5
) estimated the protective efficacy of a live attenuated influenza vaccine in a randomized controlled trial in children to be 0.89 (also see Belshe et al. (6
) and Longini et al. (7
)). Using a case definition of "upper respiratory tract illness with either fever or cough," Nichol et al. (4
) estimated the protective efficacy of a similar live attenuated influenza vaccine in a randomized controlled trial in adults to be only 0.25.
Indirect and overall effectiveness measures are obtained by comparing the risk of disease in a community that has a vaccination program with the risk in a community that has no vaccination or has a different program. In a group-randomized influenza vaccine trial for overall effectiveness (3), 30 schools were randomly assigned to receive either vaccine or placebo. The outcome was the presence of one or more respiratory illnesses or the absence of such illnesses during the epidemic period. The low estimate of overall effectiveness of a live candidate vaccine (e.g., 1 - [430/2,525]/[567/2,331] = 0.30 in one age group) could easily have resulted from the nonspecific case definition. If just half of the respiratory illness in the placebo group were not influenza, corresponding to 307 noninfluenza cases in the vaccine group, the effectiveness point estimate would be 0.59 (1 - [127/2,525]/[283/2,331] = 0.59)a factor of 2 greater.
The efficacy of a vaccine in reducing infectiousness, VEI, is usually measured by comparing the risk of transmission from a vaccinated person with the risk of transmission from an unvaccinated person (2). However, data on exposure to infection are often difficult or expensive to obtain and are inherently prone to mismeasurement.
Study designs that combine data of different levels and quality have been developed in some areas of epidemiology, such as nutritional epidemiology and cancer epidemiology. In a small subset of participants, called the validation sample, a good measurement of the exposure or outcome of interest is obtained. For each of the participants, including the validation set, a less accurate or coarser, possibly cheaper, measure is also obtained. The less accurate measure is sometimes called a "surrogate." The better measure is sometimes called the "good data" or even the "gold standard." Estimates based on just the surrogate measure would usually be biased. Estimates based on just the good data from the small validation sample would not be very precise. The general idea is that the good data in the validation set correct the bias, while the larger main study increases precision.
In a previous article (8), we suggested that these methods could be extended to improve the design and analysis of studies of infectious disease, particularly in vaccine evaluation. In this paper, we show how studies with validation sets can produce more accurate and efficient estimates in vaccine field studies. We discuss challenges posed by infectious diseases in the use of currently available methods and call for more methodological developments.
![]() |
STUDIES WITH VALIDATION SETS |
---|
Many statistical methods are available for analyzing studies with validation sets (12, 15
23
). Usually one starts with a parametric model, such as a likelihood model or an estimating equation, for the people on whom good data are available. Then one uses some method to combine the people on whom coarser data are available into the analysis. Full likelihood or Bayesian approaches usually model the relation of the coarser data to the more accurately measured data and build the model into the analysis. However, the relation between the good measure and the less accurate measure is not of any scientific interest. If the model relating the two measures is wrong, the analysis can be quite biased.
Semiparametric methods either use a nonparametric method to estimate the relation or do not estimate it all. By avoiding parametric specification of the relation between the good covariate and the surrogate, semiparametric methods avoid the bias that results from misspecifying the relation. An example is the semiparametric mean score method for outcomes (24) and for covariates (23
). In the case of covariates or exposure variables, the score contribution (i.e., the derivative of the log likelihood) for each main study member on whom only coarse exposure data are available is estimated from the average score contributions of the validation sample members with the same observed covariate and outcome values. The mean score approach for outcomes is similar in that a surrogate outcome is measured in everyone, while the accurate outcome is measured only in the validation sample. The semiparametric efficient methods of Robins et al. (20
, 21
) also avoid nonparametric estimation of the missing covariate distribution. However, the semiparametric efficient methods extract further information from people who are not in the validation set.
Several approaches have been developed for using validation sets for outcome data (15, 25
27
). In the survival setting, much work related to missing or mismeasured covariates has been conducted (28
31
), but less work has been done for misclassified outcomes.
![]() |
USING VALIDATION SETS FOR OUTCOMES |
---|
![]() |
Suppose, however, that we do not confirm suspected cases by culture. The ascertained cases of influenza-like illness will possibly include many cases that are not influenza but illnesses caused by other viruses, such as respiratory syncytial virus or parainfluenza. We will call such illnesses "noninfluenza." The term "influenza-like illness" captures both the true influenza cases and the noninfluenza cases. Exactly what the terms include will depend on the definitions used in any particular study.
Suppose that z1 and z0 are the numbers of noninfluenza cases in the vaccinated and unvaccinated groups, respectively. Then the total number of influenza-like illnesses in vaccine group v, v = 0, 1, is wv = zv + yv. The efficacy estimate based on the total number of influenza-like illnesses would be
![]() |
Consider the example shown in table 1. Our estimate of VES based on the true influenza cases would be
![]() |
![]() |
|
![]() |
A simple adjustment using a validation set |
---|
We denote the sampling fraction as v, the number of influenza-like cases sampled for the validation set as rv, and the number of culture-confirmed influenza cases as cv, v = 0, 1. We estimate the proportion
v of the influenza-like cases that are true influenza from the ratio of the number of culture-confirmed cases to the total number of influenza-like cases in each validation set, i.e.,
, v = 0, 1. This estimated proportion is used to adjust the number of influenza-like illnesses in each vaccine arm to estimate the number of true influenza cases.
Suppose that the sampling fractions are 1 = 0.20 and
0 = 0.10 in the vaccinated and unvaccinated groups, respectively. Then we would expect to sample r1 = 0.20(60) = 12 and r0 = 0.10(150) = 15 influenza-like cases for the vaccinated and unvaccinated validation samples, respectively. We would expect 10/60 of the cultured vaccinated cases to be true influenza, or c1 = 2 of the r1 = 12 cases in the validation sample. We estimate
. Similarly, we would expect 100/150 of the cultured unvaccinated influenza-like cases to be culture-confirmed influenzathat is, c0 = 10 of the r0 = 15 cases in the validation sample. We estimate
. We then multiply the observed number of all influenza-like illnesses in each group by the estimated proportion of true influenza to obtain
![]() | (1) |
![]() |
![]() |
The simple adjustment corrects for the bias resulting from using influenza-like illness as the outcome without our having to culture every suspected case. The main penalty in using the validation sample rather than culturing everyone is the increased uncertainty in the estimate. The variability of the estimate obtained using a validation sample depends on the size of the validation set. In this example, if the sampling fraction in each group were doubled, the approximate 95 percent confidence interval would decrease to (0.74, 0.96).
The degree of attenuation of the VES estimates from using the nonspecific case definition depends on the ratio of true disease to background nonspecific disease. In the above example, if instead of 50 there had been 100 noninfluenza cases in each group, the estimate based on all influenza-like illness would have been
![]() |
![]() |
Time-varying incidence rates |
---|
For example, suppose that we group the influenza-like cases within small time intervals , such as 1 week: (t
- 1, t
],
= 1, ..., T. If the influenza epidemic or vaccine study lasts 12 weeks, then T = 12. We also group the validation samples rv
within time intervals. Then we estimate the proportion
v
of true influenza cases among the influenza-like illnesses in each vaccine group v, v = 0, 1, from the validation samples in each time interval
,
= 1, ..., T; that is,
. We multiply the number of influenza-like illnesses ascertained in each week wv
by the estimated {
v
} for that time interval to obtain an adjusted estimate of the number of influenza cases in each interval. Summing over the adjusted estimates of the number of true influenza cases in each interval, we obtain an adjusted estimate of the total number of influenza cases in each group during the study. From this, we estimate the incidence proportion of true influenza in each vaccine group, and from that, VES,IP,v:
![]() | (2) |
Figure 1 shows the results of 100 simulations for estimating vaccine efficacy based on 1) true influenza, 2) the use of this simple validation set approach, and 3) all influenza-like illness. The influenza epidemic in this example lasted for 12 weeks. The expected incidence in children varied weekly as (0.014, 0.024, 0.034, 0.05, 0.06, 0.055, 0.05, 0.044, 0.038, 0.024, 0.015, 0.01). The expected incidence rate of noninfluenza was set to 0.02 per week. The expected weekly incidence rates of influenza and noninfluenza were each multiplied by an independent uniform random number between 0.85 and 1.15. Since both noninfluenza incidence and true influenza incidence were multiplied by random numbers, the ratio of true influenza to noninfluenza varied among simulations. The set vaccine efficacy was VES = 0.90 with a multiplicative (leaky) effect. In each week, we sampled 0 = 0.25 and
1 = 0.40 of the influenza-like illnesses in the unvaccinated children and the vaccinated children, respectively.
|
![]() |
Validation sets in community trials |
---|
Many features complicate community-based vaccination studies. Chief among them is the comparability of the communities included in the study with respect to the baseline incidence and the background incidence of any disease included in a nonspecific case definition. Even if the communities are comparable, however, a nonspecific case definition can attenuate the estimates of indirect and overall effects.
In figures 2 and 3, we present results of 100 simulated estimates of the indirect effects of vaccinating 50 percent of the children in one community as compared with another community without vaccination. This scenario is similar to that depicted in figure 1, with 10,000 people in each population, half children and half adults. The incidence rate of true influenza in adults is only half that in children, while the incidence rate of noninfluenza in adults is the same as that in children. The baseline incidences of true influenza and background noninfluenza are multiplied by random numbers between 0.85 and 1.15, so the baseline incidences in the two comparison communities are similar but not identical. To estimate indirect effects in children (figure 2), one compares the incidence proportion among unvaccinated children in the community that has the vaccination program with the incidence proportion among (unvaccinated) children in the community without vaccination. A similar comparison is made among the adults (figure 3), all of whom are unvaccinated. We have set the indirect effects to 0.25. The top histograms of estimates based on ascertainment of all true influenza cases in children and adults are centered around 0.25, the set value. However, if we use all influenza-like illnesses, the estimates are much lower (bottom rows). The histogram is centered around 0.14 in children and 0.10 in adults. Using the simple time-varying adjustment with validation sets that was described above (see previous section), the adjusted indirect effect estimates are once again centered more closely around 0.25.
|
|
![]() |
USING VALIDATION SETS FOR EXPOSURE TO INFECTION |
---|
With a focus on the potential for improving estimation of VEI, Golm et al. (37) assumed that only partnerships were included in the validation sample. In the validation partnerships, information on sexual contacts was assumed to be gathered without error. Thus, an easy, coarse measure of exposure consisted of each partnership's classifying their number of within-partnership contacts as either high or low (Hi/Lo). Semiparametric analytical methods (22
, 23
) were then applied. A surrogate M was also assumed as measured from each partnership's making some guess at their number of sexual contacts.
Figure 4 illustrates the potential for improving estimates of VEI. The histograms shown are from 200 simulations of a human immunodeficiency virus vaccine trial with 4,000 primary trial participants and 2,000 with steady partners. The sampling fraction was 0.20 of the partnerships for the validation set. Vaccine efficacy was set to VES = 0.4 and VEI = 0.6. (For more details, see Golm et al. (37).) The top histogram presents estimates based on participants and partners for whom complete, good data are available (complete cases). The estimates for VEI are quite variable, since there is little information available. In the next two histograms, the two semiparametric approaches provide much more precise estimates than the complete case estimates. These methods incorporate the information from the main study on people for whom only coarse exposure data are available. The surrogate M (fourth histogram) actually performs fairly well. However, using the coarse data based on Hi/Lo alone yields a very biased, though precise, estimate of VEI (bottom histogram). The problem of bias is overcome with the use of the validation set.
|
![]() |
DISCUSSION |
---|
Several challenges to use of validation sets are posed by the infectious disease setting. In the simple, time-invariant influenza example presented above, existing methods could be applicable. However, the rapidly time-varying incidence rates of some infectious diseases present new problems. The probability that any suspected case is a case of the disease of interest changes rapidly over time. Methods employed must take this rapid time evolution into account. A person might have more than one event of misclassified disease during a study (4143
), but generally a person would have only one case of the disease of interest. This raises issues related to the validation sampling scheme. The problem with sampling individuals when they sometimes present with influenza-like illness and sometimes do not is that it is possible to miss sampling them when they have true influenza. Presumably, then, when they presented again, they actually would no longer be in the risk set for having influenza. This problem could be avoided by selecting people for the validation set before the study begins and culturing them each time they present with illness fitting the nonspecific case definition. Such issues require further examination.
Other concerns in selection of the validation sample are common to the study of noninfectious diseases. The validation set may not be internal to the actual study but may be drawn from some other external population. For example, influenza epidemics are often monitored by culturing people with suspected influenza cases once the season has begun. The samples so cultured are usually convenience samples. Whether a physician decides to take a culture can be heavily influenced by his or her belief as to whether the person has true influenza. If such convenience samples were used uncritically to adjust vaccine efficacy or effectiveness estimates, the results could be very biased. However, methods for using such convenience samples could be developed. If a physician knows a person's vaccine status, it might affect whether he or she takes a sample. People who are vaccinated may have less serious illness and may tend not to visit a physician or report symptoms. Optimal sampling strategies also have yet to be explored in this context. Efficient methods will probably vary the probability of being selected into the validation set with time, as well.
Other problems, such as whether the good measure of outcome is a gold standard or itself is prone to mismeasurement, need further examination. The probability of obtaining a positive culture may depend on the vaccination status of an individual, because vaccination could shorten the period in which a positive culture can be obtained or could reduce the shedding of the infectious agent so that the culture is less likely to be positive. The choice of the nonspecific case definition is also important in determining the ratio of true cases of interest to background cases. Similar problems in using validation sets for exposure to infection remain to be solved. Of particular importance is the fact that there is really no gold standard for exposure to infection in any setting. These and many other issues could be fruitfully examined to improve vaccine efficacy and effectiveness studies.
Analytical methods for combining participants with different levels of data on exposure to infection could also be used in new approaches for estimating VES. In vaccine trials, the primary analysis generally uses one of the unconditional estimates of VES. Often information on contact with and exposure to infection is available on some of the participants, more by happenstance than by design. An example might be a pertussis vaccine trial in which information on household exposure to infection is available for some of the participants but no exposure is observed for most participants. Until now, the subset of individuals for whom exposure information was available was analyzed in a second analysis to obtain VES and, less often, VEI estimates based on the transmission probability or secondary attack rates. However, methods could be developed to incorporate the different levels of information into a single analysis to improve estimation of VES as well as VEI.
In this paper, we have considered separately the use of validation sets for outcomes and for exposure to infection. However, it would be possible for studies to use validation sets for both. Validation sets could be used in other infectious disease studies as well. In malaria studies, exposure to infection is measured by capturing mosquitoes. Validation sets for exposure to malaria could be useful in studies on developing immunity. The potential for using validation sets in infectious disease field studies has just begun to be explored. There is room for many new developments to meet the special challenges of studying infectious diseases.
![]() |
ACKNOWLEDGMENTS |
---|
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|