Use of Multiple Imputation to Correct for Nonresponse Bias in a Survey of Urologic Symptoms among African-American Men

Jeremy M. G. Taylor1, Kristine L. Cooper1, John T. Wei2, Aruna V. Sarma2, Trivellore E. Raghunathan1,3 and Steve G. Heeringa3

1 Department of Biostatistics, University of Michigan, Ann Arbor, MI.
2 Department of Urology, University of Michigan, Ann Arbor, MI.
3 Institute for Social Research, University of Michigan, Ann Arbor, MI.

Received for publication September 17, 2001; accepted for publication June 19, 2002.


    ABSTRACT
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
The Flint Men’s Health Study is an ongoing population-based study of African-American men designed to address questions related to prostate cancer and urologic symptoms. The initial phase of the study was conducted in 1996–1997 in two stages: an interviewer-administered survey followed by a clinical examination. The response rate in the clinical examination phase was 52%. Thus, some data were missing for clinical examination variables, diminishing the generalizability of the results to the general population. This paper is a case study demonstrating the application of multiple imputation to address important questions related to prostate cancer and urologic symptoms in a data set with missing values. On the basis of the observed clinical examination data, the American Urological Association Symptoms Score showed a surprising reduction in symptoms in the oldest age group, but after multiple imputation there was a monotonically increasing trend with age. It appeared that multiple imputation corrected for nonresponse bias associated with the observed data. For other outcome measures—namely, the age-adjusted 95th percentile of prostate-specific antigen level and the association between urologic symptoms and prostate volume—results from the observed data and the multiply imputed data were similar.

bias (epidemiology); Blacks; imputation; nonresponse; urology

Abbreviations: Abbreviations: AUABS, American Urological Association Bother Score; AUASS, American Urological Association Symptoms Score; MCAR, missing completely at random; PSA, prostate-specific antigen.


    INTRODUCTION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Prostate cancer and urologic symptoms are major health concerns for men over 40 years of age. The incidence of prostate cancer in the United States has increased significantly over the last 15 years; however, much of this increase has been attributed to the use of the prostate-specific antigen (PSA) test. It is also well known that African-American men have both a higher incidence of the disease and higher mortality from the disease compared with White men in the United States (1), and it has been suggested that there is a difference between African-American men and White men in the age-specific reference range for PSA (2). Data from a hospital-based cohort (3) suggested that the normal range of PSA values may be higher for African-American men than for White men. Lower urinary tract symptoms are exceedingly common among older men and have a major impact on health-related quality of life; moderate to severe symptoms occur in 28 percent of men over age 70 (4). These symptoms can lead to more worry and greater interference with daily activities (5).

The Flint Men’s Health Study is an ongoing population-based random survey of African-American men in Flint, Michigan. Major aims of the study are to define age-adjusted normal ranges for PSA levels in African-American men without prostate cancer and to define the prevalence of lower urinary tract symptoms. To our knowledge, it is the first population-based study to evaluate urinary symptoms among African-American men. Details on the study design and preliminary results have been published elsewhere (68).

Data collection was conducted in two stages: first an interview questionnaire and second, for eligible men, a follow-up clinical examination that included a biopsy in a selected subset of men. There was a high response rate at the interview stage (86 percent) but a lower response rate (52 percent) at the clinical examination stage. The low response rate in the clinical examination stage was viewed as a potential problem for interpretation of some of the study’s results. For example, a specific quantity of interest is the prevalence of lower urinary tract symptoms, as measured by the American Urological Association Symptoms Score (AUASS). Results for this measure in the clinical examination stage showed a trend towards increasing AUASS with age but a surprising decrease in the highest age group (8). This finding may have been a consequence of nonresponse bias.

It is common to have missing data in epidemiologic studies. Gaps in the data can lead to biased assessment and misleading interpretation of the data collected. Statistical theory associated with missing data has been characterized elsewhere (9), and this is an active area of research. One generally applicable approach to analyzing data with missing values is multiple imputation (10, 11). Use of multiple imputation for assistance in the analysis of public health and epidemiologic data is becoming more common (1216). With the multiple imputation method, missing values are "filled in" in an appropriate way consistent with the observed data. The procedure is repeated multiple times to create multiple complete data sets. Each data set is then analyzed separately, and the results are combined to obtain statistical inferences about quantities of interest. The quantities of interest can be measures of association, such as regression coefficients or odds ratios, or summary statistics for a particular variable, such as the mean value or 95th percentile. The theory and implementation of multiple imputation and the conditions for its validity have been described in a paper by Rubin (11) and in many subsequent articles.

In the Flint Men’s Health Study, values for clinical examination variables for the men who did not have a clinical examination can be regarded as missing data. Similarly, biopsy results for men who did not undergo biopsy are missing data. For each missing data point, a missing value can be conceptualized as the value that would have been obtained if that variable had been measured. If the missing values were known, the data set would be complete, and most analyses would be relatively straightforward to perform. Since this is a population-based sample, an inference could then be validly generalized to the population from which the sample was chosen. We undertook multiple imputation of missing data in the Flint Men’s Health Study to correct for potential nonresponse bias in our interpretation of the normal range of PSA levels and self-administered AUASS.


    MATERIALS AND METHODS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Study design
The Flint Men’s Health Study was designed as a two-stage population-based study of a random sample of men aged 40–79 years (7). Participants were enrolled in 1996 and 1997. Of the 943 age-eligible men identified in the sample, 819 (86.8 percent) completed the interview. The interview consisted of a lengthy questionnaire regarding demographic information, medical history, and family history of cancer. Specific questions were asked about urologic symptoms (AUASS), prior surgeries, and prostate cancer. Men who had had previous urologic surgery or a known diagnosis of prostate cancer were excluded from clinical follow-up. The remaining 730 men were invited to a second stage of the study which consisted of blood drawing for a PSA test and a full clinical examination. The clinical examination included a digital rectal examination, an assessment of urine flow rate, ultrasound assessment of prostate volume, and self-administration of the AUASS and American Urological Association Bother Score (AUABS) scales. Of the 730 eligible men, 381 participated in the clinical examination phase of the study, for a response rate of 52 percent. Selected men who had a PSA level greater than 4.0, an abnormal digital rectal examination, or an indication of abnormality upon ultrasound were invited to undergo prostate biopsy. Forty-eight men had a biopsy performed as part of the study.

Study variables
PSA level, self-administered AUASS, and biopsy result were the primary variables of interest in this analysis. However, the multiple imputation procedure must simultaneously consider other variables that might be associated with the three primary variables and variables that might be associated with the probability of participating at the clinical stage or the biopsy stage of the study. For example, questionnaire AUASS clearly will be strongly correlated with self-administered AUASS and therefore will be very useful for imputing the missing self-administered AUASS data. We performed an investigation of factors associated with nonparticipation in the clinical phase of the study (6, 7). We found that age, family history of prostate cancer, alcohol use, and AUASS were associated with participation in the clinical stage of the study.

Table 1 describes the variables used in the multiple imputation analysis, including the type of variable, its range, and the number of observed data points. The 48 observed biopsies (11 positive, 37 negative) in the study have been augmented by biopsy results from nine additional men who reported having a negative biopsy during the past year on the questionnaire. In our analyses, a person was regarded as having an indication for biopsy if the PSA level was greater than 4, the digital rectal examination was not normal, or the ultrasound results were viewed as suspicious.


View this table:
[in this window]
[in a new window]
 
TABLE 1. Variables used in the multiple imputation analysis, Flint Men’s Health Study, 1996–1997
 
Primary endpoints
The primary endpoints for which results are presented here are, for men who were biopsy-negative, 1) mean AUASS (self-administered), 2) 95th percentile of PSA level, 3) proportion of men who had a normal digital rectal examination, and 4) regression coefficient 1) of prostate volume in the model AUASS = ß0 + ß1volume + ß2age. This regression coefficient is designed to assess the association between prostate volume and AUASS, controlling for age. The final endpoint is 5) the proportion of the population who were biopsy-positive.

Endpoints 1–4 above are generalizable to a population of African-American men aged 40–79 years who have had no previous urologic surgery, have had no prior diagnosis of prostate cancer, and would have no evidence of prostate cancer from a biopsy. Endpoint 5 is generalizable to a population of African-American men who have had no prior urologic surgery and have no prior known diagnosis of prostate cancer.

Multiple imputation procedure
In multiple imputation, the missing values are drawn from an appropriate distribution that characterizes the conditional relation of the imputed variables to other variables. Of course, the imputed values for any subject are not real values and have no interpretation. They are used simply as statistical tools with which to effectively use other nonmissing variables from that subject to make an inference about the quantity of interest. The procedure of drawing missing values from the distribution is repeated M times. Because the missing values are drawn from a distribution, there will be a range of values imputed for each missing value, with this variation appropriately reflecting the uncertainty about that value. After imputation, each of the M completed data sets is analyzed separately, and the results are combined. The Bayesian theoretical underpinnings of the method (11) require a statistical model for the joint probability distribution of all of the variables. On the basis of this model, each missing value is drawn from an appropriate distribution. Let Q denote the quantity of interest—for example, the mean AUASS; let Xobs denote observed data and Xmis the missing data; and let {theta} denote the parameters in the model for X = (Xobs, Xmis). Inference about Q can be represented by the posterior distribution of Q given Xobs. Technical details on the theory dictating how the values of Xmis are imputed are given by Little and Rubin (9).

For each of the completed data sets, the inference for Q can be summarized by an estimate and a standard error; denote these m and SEm, m = 1, ..., M. Then the final estimate for Q is

,

and the final variance is a sum of between- and within-imputation variances given by

. (1)

The implementation of multiple imputation in this study is based on the procedure described by Raghunathan et al. (17) using software called IVEware (18). Multiple imputation requires that the joint distribution of all of the variables which have missing values be specified; this is difficult when the number of variables is large and the variables are of continuous, categorical, and mixed types. In this procedure, the joint distribution is approximated by a series of conditional distributions of each variable conditioned on all other variables. The conditional distribution of each continuous variable is given by a standard linear regression model. Logistic regression is used for binary variables. For count variables, Poisson regression is used. For categorical variables with more than two response categories, a polytomous regression model is used. A mixed variable is one which is either zero or positive, and if it is positive it has a continuous distribution. For imputing a mixed variable, first the zero/nonzero status is imputed using logistic regression and then, conditional on its being nonzero, normal linear regression is used to impute a continuous value.

The IVEware procedure iterates through all of the variables. For each variable to have data imputed, the regression model is fitted to the current values of observed and imputed data, with that variable designated as the response variable and the others included as the covariates. The model fitting gives estimates and the covariance matrix for the parameters {theta}. A value of {theta} is drawn from a multivariate normal approximation to the posterior distribution of {theta}. Then a new value of the missing data is drawn from the model specified by the regression equation, using the explanatory variables and the drawn value of {theta}. This is repeated with each variable considered as the response variable, in sequence. The whole sequence is repeated pM times. With every pth set of sequences, the imputed values are saved, giving M completed data sets. The value of p needs to be large enough that the imputed values are essentially uncorrelated, and M is typically 5 or greater, with larger values being preferred if there is a substantial amount of missing data. For these analyses, we used p = 20 and M = 20.

For binary variables for which there is a substantial amount of missing data, specifically for the "biopsy result" and the "indicator for biopsy needed" variables, two refinements of the algorithm are used. An importance sampling scheme is used to draw {theta} from its posterior distribution (12), and the number of explanatory variables was limited to the best six.

Within each age category, the final variance of the multiple imputation estimate is a sum of the within- and between-imputation variance components. For the mean AUASS, the within-imputation standard error is SD/ , and for a percentage, the within-imputation standard error is given by , where n is the sample size within each age group. The within-imputation standard error for the 95th percentile of PSA is calculated from the formula

, (2)

where P = 0.95 and f (.) denotes the probability density function, which was estimated from the data.

Survey weights
The design of the Flint Men’s Health Study included oversampling of older individuals. The sampling weights for the four age categories 40–49, 50–59, 60–69, and 70–79 years were 4.45, 2.96, 2.34, and 1.00, respectively. The weights reflect age-specific adjustments for the population-based sampling probabilities and minor corrections for the small differences in age-specific rates of response to the interview questionnaire, which were slightly lower for men aged 60–79 years. These weights are important because they must be taken into account in any overall estimates and measures of uncertainty. For example, the overall average AUASS is a weighted average of the age-category-specific mean AUASS values. The standard errors accounting for the sampling weights were computed using the Taylor series approach when estimating weighted means and proportions and the "jackknife" approach when estimating coefficients in the regression models (19) as implemented in the IVEware software. For the overall 95th percentile, a weighted mean and variance were calculated to approximate the density in equation 2.


    RESULTS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Table 2 shows the mean AUASS prior to imputation for the four age categories for both variables. We see a trend towards increasing AUASS with age for the questionnaire version but, surprisingly, a decrease in the highest age group for the self-administered version. Examination of the questionnaire AUASS for men who returned and had AUASS measured at the clinic visit and men who did not shows the reason for this nonresponse bias: There was a tendency for those with more symptoms in the oldest category not to participate in the clinical phase of the study.


View this table:
[in this window]
[in a new window]
 
TABLE 2. American Urological Association Symptoms Score (mean values) from data observed in the questionnaire and obtained at the clinical examination, Flint Men’s Health Study, 1996–1997
 
Twelve different imputation models were considered. All the models were variations of the same basic theme. Model 1 in table 3 is the base model in which some variables are considered continuous, others categorical, and others mixed in the regression equations that generate the imputations. The 11 variations of this model were to alter the functional form of some of the regression models through the inclusion of extra variables, interactions or quadratic terms, different transformations, or categorizations of continuous variables. For example, in model 2, each time a regression equation involves AUASS or AUABS, as either the response or the covariate, the data are changed to the log of that value plus 1. The purpose of including several different models was to assess the sensitivity of any conclusions to variations in the choice of statistical model.


View this table:
[in this window]
[in a new window]
 
TABLE 3. Imputation models considered in the sensitivity analysis, Flint Men’s Health Study, 1996–1997
 
The results of the imputation procedures for self-administered AUASS among biopsy-negative men are given in table 4. For all analyses following use of the various imputation models, mean AUASS increases with age. This differs from the conclusion that would be reached from the observed data.


View this table:
[in this window]
[in a new window]
 
TABLE 4. Results of multiple imputation for self-administered American Urological Association Symptoms Score (mean values) among biopsy-negative men, Flint Men’s Health Study, 1996–1997
 
The results for 95th percentile of PSA are given in table 5. The table shows increasing PSA with age for all imputation models, with consistent age-specific estimates across the 12 imputation models. The main difference from the observed data is that the 95th percentile is lower for the imputed results than for the observed data in 70- to 79-year-olds. The reason is that the observed-data result is determined from this small sample size of 57 biopsy-negative men by the value of the third highest PSA, and thus it will be quite sensitive to typical variations in extremes of the data. The results from the multiple imputation procedure are based on a larger sample size and are the average of 20 estimates and thus will be less sensitive to outliers. Furthermore, the observed-data result excludes men who had a positive biopsy, but it does not exclude men who had a high PSA level but did not have a biopsy performed.


View this table:
[in this window]
[in a new window]
 
TABLE 5. Results of multiple imputation for 95th percentile of prostate-specific antigen among biopsy-negative men, Flint Men’s Health Study, 1996–1997
 
The results for the proportion of biopsy-positive men among the 730 men eligible are given in table 6. With the slight exception of models 9 and 12, the results consistently suggest that the proportion of men who would show positive if they underwent biopsy is 3 percent, with a standard error of approximately 1 percent. This is substantially less than the 17 percent in the observed data. In addition, the results (not shown) indicate that the proportion of men who are biopsy-positive increases from approximately 1 percent in 40- to 49-year-olds to 8–14 percent in 70- to 79-year-olds. Among all men with a PSA level greater than 4, the estimated percentage of biopsy-positive from model 1 is 30 percent (standard error, 9). Table 6 also shows the proportion of biopsy-negative men who have a normal digital rectal examination. The proportion from all imputation models is similar to the results from the observed data. In addition, table 6 shows the estimated coefficients and standard errors for prostate volume in the regression model designed to predict AUASS controlling for age. From the observed data, prostate volume is not a significant predictor of AUASS. The same is the case for all of the multiply imputed data sets, suggesting that this conclusion is robust and not influenced by the nonresponse bias in the study.


View this table:
[in this window]
[in a new window]
 
TABLE 6. Comparison between observed data and results of multiple imputation for overall proportion of participants who are biopsy-positive, overall proportion with a normal digital rectal examination, and overall association between American Urological Association Symptoms Score and prostate volume, controlling for age, Flint Men’s Health Study, 1996–1997
 

    DISCUSSION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Missingness of data is a common problem in many epidemiologic and clinical follow-up studies, and it can lead to biased and inefficient estimates. Multiple imputation is an appealing and convenient approach for the analysis of such data, and under appropriate conditions it provides valid inferences for quantities of interest. There are other methods that could be used to adjust for bias caused by missing data. For example, weighting adjustment (14) could be used if there were no missing values in the questionnaire data. These methods are likely to be effective in reducing nonresponse bias, but multiple imputation has some advantages, such as its applicability to general patterns of missing data and potentially its greater efficiency and more accurate standard errors. We specifically note that multiple imputation can be used when there is missing information in both the intended outcome and covariates.

In this paper, we implemented multiple imputation using IVEware software. There are a number of programs available for multiple imputation, some of which were reviewed by Horton and Lipsitz (20) and can be found on the World Wide Web site http://www.multiple-imputation.com. There is no single software product that can handle all designs and data types. IVEware was particularly convenient for our data because it can be applied when there is a large number of continuous and categorical variables, it can accommodate general patterns of missingness, and it has a procedure for incorporating the survey sampling weights into the final analysis.

The terminology of missing data (9) classifies the missingness pattern into three types: missing completely at random (MCAR), missing at random, and nonignorable missingness. The MCAR pattern occurs when the probability of missingness does not depend on the value of the data. In the context of the Flint Men’s Health Study, this would occur if the men who did not participate at the clinical examination stage were a random sample of all eligible men from the questionnaire stage. When MCAR does occur, analysis of the observed data is valid, although it may not be efficient and techniques such as multiple imputation could improve the efficiency. From table 2, it is clear that MCAR is not valid for the Flint Men’s Health Study.

The pattern "missing at random" occurs when the probability of missingness is dependent on the observed data but not on the unobserved data. Under these conditions, simple analysis of the observed data can give an invalid statistical inference. For example, the proportion biopsy-positive from the observed data is clearly not a valid estimate of the population proportion. However, likelihood-based or model-based methods will give a valid inference provided that the models are correctly specified and the variables that determine the probability of missingness are included in the model. Multiple imputation is an example of a model-based method.

Nonignorable missingness is where the probability of missingness depends on the value of the unobserved data. In this case, likelihood-based methods are not valid unless a model for the missingness mechanism is also specified. In general, there is no information in the observed data with which to check the appropriateness of the missingness model.

Previous analysis (6, 7) of data from the Flint Men’s Health Study indicated that the probability of nonparticipation in the clinical examination stage was associated with the questionnaire variables age, AUASS, and number of alcoholic drinks per day, among other factors. There was a lower participation rate among older men and those who had urologic symptoms or were already seeing a doctor on a regular basis. A possible interpretation is that younger and healthier men participated at a higher rate because they viewed the clinical examination as beneficial, whereas older participants tended to be already under the care of a doctor and saw less benefit in the clinical examination. Because factors associated with participation in the clinical examination were obtained at the questionnaire stage and included in the model, it is plausible that the missingness mechanism is "missing at random," for which multiple imputation is valid.

As a sensitivity analysis, we have presented the results from 12 different imputation models. Numerically the results from the models differ, but with a few exceptions, the differences are slight, giving some reassurance as to the robustness of the conclusions. If one wants to select a best model, one should include in it variables that are associated with the quantities of main interest and variables that are associated with the probability of missingness. Standard model assessment and checking procedures can be used to choose a specific functional form for the model. However, it may not be possible to decide on the functional form of the best model from the observed data; in these situations, it is important to perform a sensitivity analysis to assess the robustness of the conclusions. The results for model 12 for the percentage biopsy-positive are different because they are based on an imputation model that excludes the "indication for biopsy" variable. Similarly, in model 9, the definition of "indication for biopsy" was changed. This variable is known to be strongly associated with the probability of having a missing biopsy result. This illustrates the importance of including in the imputation model variables that are known to be associated with the missingness process (21).

In the current study, multiple imputation was applied to improve efficiency for estimating appropriate age-specific reference ranges. The Flint Men’s Health Study was established to determine the age-specific reference ranges for PSA among African-American men, but the sample has also been used to evaluate the association between aging and lower urinary tract symptoms. The rationale for applying community-based cohorts to the derivation of normative reference ranges is minimization of the selection bias inherent in the use of clinical samples. It is apparent from table 2 that nonresponse bias affected the interpretation of AUASS from the clinical stage. Following multiple imputation, there is a clear increasing trend in average AUASS with age. The various imputation models show small differences in the average AUASS, but they all show increasing trends with age. The differences in average AUASS are all within the uncertainty limits of one another, so this is not a cause of great concern. The fact that all models showed increasing AUASS with age suggests that this conclusion is quite robust.

The results for 95th percentile of PSA are quite consistent across imputation models. A similarly designed study of White men in Olmsted County, Minnesota (22), estimated the 95th percentile of PSA level to be 2.0, 2.9, 5.8, and 6.2 in persons aged 40–49, 50–59, 60–69, and 70–79 years, respectively. These numbers are quite similar to our results, and thus we do not find evidence for a difference in the normal range of PSA values between White and African-American men.

In summary, multiple imputation is an attractive and feasible approach for analyzing data sets with missing values and for assessing and potentially correcting for nonresponse bias. This analysis demonstrates the utility of multiple imputation in a large population-based study of prostate disease.


    ACKNOWLEDGMENTS
 
This research was supported by grant P50 CA69568 from the National Cancer Institute.


    NOTES
 
Reprint requests to Prof. Jeremy M. G. Taylor, Department of Biostatistics, University of Michigan, 1420 Washington Heights, Ann Arbor, MI 48109 (e-mail: jmgt{at}umich.edu). Back


    REFERENCES
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Stanford J, Stephenson R, Coyle L, et al. Prostate cancer trends 1973–1995. Bethesda, MD: National Cancer Institute, 1999. (NIH publication no. 99-4543).
  2. Sawyer R, Berman J, Borkowski A, et al. Prostate-specific antigen in black men. (Letter). Lancet 1996;347:1329.
  3. Morgan TO, Jacobsen SJ, McCarthy WF, et al. Age-specific reference ranges for serum prostate-specific antigen in black men. N Engl J Med 1996;335:304–10.[Abstract/Free Full Text]
  4. Chute C, Panser L, Girman C, et al. The prevalence of prostatism: a population-based survey of urinary symptoms. J Urol 1993;150:85–9.[ISI][Medline]
  5. Girman C, Epstein R, Jacobsen S, et al. Natural history of prostatism: impact of urinary symptoms on quality of life in 2115 randomly selected community men. Urology 1994;44:825–31.[ISI][Medline]
  6. Cooney KA, Strawderman MS, Wojno K, et al. Age-specific distribution of serum prostate-specific antigen in a community-based study of African-American men. Urology 2001;57:91–6.[ISI][Medline]
  7. Heeringa SG, Alcser KH, Doerr K, et al. Potential selection bias in a community-based study of PSA levels in African-American men. J Clin Epidemiol 2001;54:142–8.[ISI][Medline]
  8. Wei JT, Schottenfeld D, Cooper K, et al. The natural history of lower urinary tract symptoms in black American men: relationships with aging, prostate size, flow rate and bothersomeness. J Urol 2001;165:1521–5.[ISI][Medline]
  9. Little RJ, Rubin D. Statistical analysis with missing data. New York, NY: John Wiley and Sons, Inc, 1987.
  10. Rubin D. Multiple imputation after 18+ years. J Am Stat Assoc 1996;91:473–89.[ISI]
  11. Rubin D. Multiple imputation for non-response in surveys. New York, NY: John Wiley and Sons, Inc, 1987.
  12. Crawford SL, Tennstedt SL, McKinlay JB. A comparison of analytic methods for non-random missingness of outcome data. J Clin Epidemiol 1995;45:209–19.
  13. Greenland S, Finkle WD. A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol 1995;142:1255–64.[Abstract]
  14. Kessler RC, Little RJ, Groves RM. Advances in strategies for minimizing and adjusting for survey nonresponse. Epidemiol Rev 1995;17:192–204.[ISI][Medline]
  15. Heitjan DF. Annotation: what can be done about missing data? Approaches to imputation. Am J Public Health 1997;87:548–50.[ISI][Medline]
  16. Zhou XH, Eckert GJ, Tierney WH. Multiple imputation in public health research. Stat Med 2001;20:1541–9.[ISI][Medline]
  17. Raghunathan TE, Lepkowski JM, Van Hoewyk J, et al. A multivariate technique for multiply imputing missing values using a sequence of regression models. Surv Methodol 2001;27:85–95.
  18. Raghunathan TE, Solenberger PW, Van Hoewyk J. IVEware: imputation and variance estimation software. Ann Arbor, MI: Institute for Social Research, University of Michigan, 1998. (http://www.isr.umich.edu/src/smp/ive).
  19. Shao J, Tu D, Krickeberg K. The jackknife and bootstrap. New York, NY: Springer-Verlag New York, 1995.
  20. Schafer JL. Analysis of incomplete multivariate data by simulation. New York, NY: Chapman and Hall, Inc, 1997.
  21. Horton NJ, Lipsitz SR. Multiple imputation in practice: comparison of software packages for regression models with missing variables. Am Stat 2001;55:244–54.[ISI]
  22. Schafer JL. Analysis of incomplete multivariate data by simulation. New York, NY: Chapman and Hall, Inc, 1997.
  23. Oesterling JE, Jacobsen SJ, Chute CG, et al. Serum prostate-specific antigen in a community-based population of healthy men: establishment of age-specific reference ranges. JAMA 1993;270:860–4.[Abstract]