RE: "PROBLEMS DUE TO SMALL SAMPLES AND SPARSE DATA IN CONDITIONAL LOGISTIC REGRESSION ANALYSIS"

Beat E. Neuenschwander and Marcel Zwahlen

Swiss Federal Office of Public Health Division of Epidemiology and Infectious Diseases 3003 Berne, Switzerland

Greenland et al. (1Go) offer a critical discussion of problems arising in the conditional logistic regression model when information in the data is weak relative to background knowledge. They warn against an uncritical use of this model and recommend that "analysts need to inspect their data in detail in order to alert themselves to the possible danger [of statistical bias]..." (1, p. 537). We fully agree on this point but would like to add two remarks to their investigation.

Statistical inference, the methodology used most widely to tackle inference problems in the empirical sciences, is hampered by two major difficulties:

  1. To make probabilistic uncertainty statements about quantities of interest, statistical models are mandatory. But where do they come from, and how can uncertainty about the most appropriate model be accounted for? Model uncertainty is still a problem widely neglected in statistical applications, even though strategies to account for it are available (2Go);
  2. For a given model, there is (still) no agreement on how its parameters should be inferred from the data. Frequentists offer various estimation procedures, all of them reasonable in their own methodological domain, but none of them undisputed. A Bayesian, in contrast, adopts a different way of reasoning: Starting from his or her prior distribution, the axioms of probability imply a unique solution, the posterior distribution of the parameters. In the case of vague prior information, point and interval estimates are typically similar to their frequentist counterparts, even though their interpretation differs.

The paper by Greenland et al. (1Go) provides an instructive illustration of both problems.

With regard to point 1, the authors focus on the conditional logistic regression model. This limitation is of particular concern because it ignores model uncertainty. Hierarchical models are mentioned as an alternative, but no estimates are presented. These models might be better suited to incorporate the right degree of matching (see reference 3) and might yield more reasonable estimates.

With regard to point 2, the authors present various frequentist estimates and, finally, Bayesian estimates for two fairly vague prior distributions reflecting background knowledge. The reader gets the impression that there is considerable uncertainty about the underlying odds ratios, but even more uncertainty about what strategy to follow when analyzing sparse data problems similar to the ones presented. While the first impression is correct, the second one is homemade by the statistical community. The prevalent school (frequentism) offers various methods to obtain point estimates, standard errors, and uncertainty bounds. All of these methods seem fairly reasonable, but there exists no rational way to decide which one to use.

We think that these fundamental problems necessitate a more general and coherent approach to data analysis. An approach that addresses these problems should proceed along the following lines: 1) the analysis of different plausible models to cope with model uncertainty. Often, a best model can be singled out when information in the data is strong. Otherwise, a fully Bayesian approach using model probabilities should be taken; 2) the implementation of the Bayesian methodology, which offers the advantage of being able to incorporate relevant background knowledge and avoiding those rather arbitrary decisions about the type of method for computing point and interval estimates.

In public health, well-calibrated risk assessments are crucial, and decision-making is often the final goal. An analysis that enables the scientist to fuse all of the relevant information is certainly advantageous and is therefore desirable. Epidemiologists might benefit by adopting the Bayesian approach that combines their subject-matter knowledge with empirical data.

REFERENCES

  1. Greenland S, Schwartzbaum JA, Finkle WD. Problems due to small samples and sparse data in conditional logistic regression analysis. Am J Epidemiol 2000;151:531–9.[Abstract]
  2. Draper D. Assessment and propagation of model uncertainty. J R Stat Soc B 1995;57:45–97.[ISI]
  3. Liang KY, Zeger SL. On the use of concordant pairs in matched case-control studies. Biometrics 1988;44:1145–56.[ISI][Medline]

 

THE FIRST AUTHOR REPLIES

Sander Greenland

Department of Epidemiology UCLA School of Public Health Department of Statistics UCLA College of Letters and Science Los Angeles, CA 90095–1772

I thank Neuenschwander and Zwahlen (1Go) for their perceptive comments. The issues they raise are too deep to address here, but I have published a number of articles that discuss them. Those articles are based on works by others, most of all those by Leamer (2Go) and Good (3Go). The work by Leamer (2Go, chap. 5) is especially relevant in showing how models and modeling procedures correspond to prior distributions and in demonstrating the profound logical and scientific flaws inherent in still-standard approaches (e.g., stepwise regression). A few key points from that book are summarized elsewhere (4Go, 5Go).

One view of modeling derived from theory and empirical studies (2Go, 3Go, 6Go, 7Go) is that a model is a smoothing device to estimate various nonparametrically defined parameters such as population risk ratios. Models whose link functions (8Go) are canonical (like the logistic) or close to it work much better in this role than do other models when the data set is small or sparse. To compensate for this technical limitation on the choice of link function, one should attempt to keep the linear predictor flexible (e.g., by entering products, powers, or splines). One then, however, faces the problem of balancing the number of terms in the linear predictor against the limits of data.

Hierarchical regression (mixed or multilevel modeling) allows one to enter many more terms in the predictor than is possible with ordinary methods; it does so by using smooth, contextual constraints that limit the effective degrees of freedom of the model (9Go). These constraints reduce the small-sample and sparse-data artifacts that plague both unconditional and conditional maximum likelihood (9Go, 10Go). The additional terms also allow better uncertainty assessments than do standard model-selection techniques (9Go, 11Go).

Because hierarchical regression has both Bayesian and frequentist rationales (12Go), it offers an operationalization of the Bayesian-frequentist fusion advocated by statisticians who recognize the fallacies in mechanical or monolithic approaches to data analysis (3Go, 13GoGo–15Go). An alternative approach popular among some Bayesians is to average results over a finite set of models (16Go); this strategy may be viewed as a crude approximation to the smooth averaging attainable with in hierarchical regression (15Go, 17Go). Regardless of what methods they use, I believe that epidemiologists (and, indeed, all scientists) should understand Bayesian probability logic (18GoGo–20Go), for the harsh light such logic casts on questionable statistical methods (such as the use of "noninformative" or "reference" priors in Bayesian statistics) will encourage analyses that are more sound scientifically than the current norm.

REFERENCES

  1. Neuenschwander BE, Zwahlen M. Re: "Problems due to small samples and sparse data in conditional logistic regression analysis." (Letter). Am J Epidemiol 2000;152:688.[Free Full Text]
  2. Leamer EE. Specification searches. New York, NY: John Wiley & Sons, 1978.
  3. Good IJ. Good thinking. Minneapolis, MN: University of Minnesota Press, 1982.
  4. Robins JM, Greenland S. The role of model selection in causal inference from nonexperimental data. Am J Epidemiol 1986;123:392–402.[ISI][Medline]
  5. Greenland S. Introduction to regression modeling. Chap. 21. In: Rothman KJ, Greenland S, eds. Modern epidemiology. 2nd ed. Philadelphia, PA: Lippincott-Raven, 1998:401–32.
  6. Maldonado G, Greenland S. A comparison of the performance of model-based confidence intervals when the correct model form is unknown. Epidemiology 1994;5:171–82.[ISI][Medline]
  7. Greenland S, Maldonado G. The interpretation of multiplicative model parameters as standardized parameters. Stat Med 1994;13:989–99.[ISI][Medline]
  8. McCullagh P, Nelder JA. Generalized linear models. 2nd ed. New York, NY: Chapman and Hall, 1989.
  9. Greenland S. When should epidemiologic regressions use random coefficients? Biometrics (In press).
  10. Greenland S. Small-sample bias and corrections for conditional maximum-likelihood odds-ratio estimators. Biostatistics 2000;1:113–22.[Abstract/Free Full Text]
  11. Greenland S. Methods for epidemiologic analyses of multiple exposures: a review and comparative study of maximum likelihood, preliminary-testing, and empirical-Bayes regression. Stat Med 1993;12:717–36.[ISI][Medline]
  12. Greenland S. Principles of multilevel modeling. Int J Epidemiol 2000;29:158–67.[Abstract/Free Full Text]
  13. Box GEP. An apology for ecumenism in statistics. In: Box GEP, Leonard, T, Wu C-F, eds. Scientific inference, data analysis, and robustness. London, England: Academic Press, Inc., 1983:51–84.
  14. Rubin DB. Bayesianly justifiable and relevant frequency calculations. Ann Stat 1984;12:1151–72.[ISI]
  15. Senn S. Consensus and controversy in pharmaceutical statistics (with discussion). Statistician 2000;44:135–76.
  16. Raftery AE. Approximate Bayes factors and accounting for model uncertainty in generalized linear models. Biometrika 1996;83:251–66.[Abstract]
  17. Greenland S. Multilevel modeling and model averaging. Scand J Work Environ Health 1999;25 (Suppl. 4):43–8.[ISI][Medline]
  18. Earman J. Bayes or bust? Cambridge, MA: MIT Press, 1992.
  19. Howson C, Urbach P. Scientific reasoning: the Bayesian approach. 2nd ed. Urbana, IL: Open Court, 1993.
  20. Greenland S. Probability logic and probabilistic induction. Epidemiology 1998;9:322–32.[ISI][Medline]