1 Department of Neurology, University of California, San Francisco, San Francisco, CA.
2 Division of Biostatistics, School of Public Health, University of California, Berkeley, Berkeley, CA.
3 Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, CA.
Received for publication May 2, 2001; accepted for publication May 24, 2002.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
confounding factors (epidemiology); ecology; instrumental variable; selection bias; simulation
Abbreviations: Abbreviation: OR, odds ratio.
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Ecologic studies may bypass confounding at the individual level. Even when confounding obscures a true underlying benefit of a therapy at the individual level, grouping the unit of analysis can produce more reliable results (57). For example, hospitals that use a more effective treatment more frequently should have better overall outcomes for patients treated there compared with another hospital that does not offer the treatment. Regardless of the pretreatment prognosis of those selected for the more effective therapy, outcomes in this group will be better than if the therapy were not available. Variability in hospital or regional practice provides a form of pseudo randomization.
On the basis of these concepts, we previously compared traditional surgical treatment of intracranial aneurysms with a new, minimally invasive technique in a study involving 70 academic medical centers (8, 9). Recognizing that utilization differences among hospitals were likely primarily due to practice variability rather than differences in patient prognosis (5, 6), we used the proportion of patients treated by the newer technique at a given hospital (the probability of receiving the newer treatment among patients in the analysis treated at the hospital) as the treatment variable and did not include in the model the actual treatment each patient received. Rather than discarding useful information by aggregating all data by hospital, we constructed models at the individual level: They predicted an individuals probability of in-hospital death. Because demographic variables could influence both the proportion of cases treated by the new technique at a hospital and the outcome of cases, we included these in the model. The final model was constructed like a standard multivariable analysis except that the actual treatment assignment for a patient was substituted by a grouped-treatment variable: the proportion of cases treated by the new technique at the hospital of treatment. More generally, a grouped-treatment variable is defined as the probability of an individuals receiving the treatment of interest at a specified level of aggregation. It can be calculated by determining the proportion of cases treated by the therapy of interest at any specified unit of aggregation, such as a ward, clinic, hospital, region, state, or country.
For the grouped-treatment approach to produce valid estimates of the treatment effect, several assumptions are required. The most important assumption, which we term the basic instrumental variable assumption, is that unmeasured confounders do not produce an association between pretreatment prognosis and the grouped-treatment variable. In other words, conditional on the value of the actual treatment and the modeled covariates, the grouped-treatment variable and the outcome are statistically independent. This assumption may be difficult to satisfy. For example, hospitals that use the new treatment for intracranial aneurysms may receive higher risk cases by referral or may offer other technologies and expertise that could contribute to outcome differences. Thus, the grouped-treatment approach is also limited by the possibility of unmeasured confounding.
The utility of aggregating treatment assignments in avoiding confounding has been recognized for some time by economists, who have termed the approach the instrumental variable method (10, 11), but it has been used infrequently by epidemiologists and health services researchers (12). Economists and epidemiologists tend to construct multivariable models differently, and economists rarely study dichotomous outcomes (10, 13, 14), which are frequently of interest in health studies. The grouped-treatment approach attempts to preserve a model structure familiar to health researchers. Several authors have described the mathematical basis for the grouped-treatment approach (15), most frequently in the context of illustrating the impact of omitted covariates (1618). However, the potential utility of the method as a means for evaluating treatment effects in the presence of unmeasured confounders has not been the subject of prior reports. Furthermore, the validity of the method has not been illustrated with terms and data familiar to epidemiologists. In this study, we subject the method to testing with simulated data sets in order to test its validity and determine its limitations.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Simulations
For each set of conditions tested, 1,000 patient cohorts with N individuals were simulated using S Plus (version 2000; Mathsoft Corporation, Seattle, Washington) software. We modeled a dichotomous treatment Ti, meant to represent a new therapy, received at hospital Hi, where i was unique for each individual 1, 2, 3, ..., N. Individuals were distributed evenly among 50 hospitals, defined by Hj = 1, 2, 3, ..., 50, with varying utilization of the treatment, as defined below. A confounder Ci and a covariate Bi were randomly and independently assigned a value of 0 or 1 to each patient with 50 percent probability. The confounder was related to both outcome and treatment according to the odds ratios (ORs) ORCO and ORCT, respectively. The covariate was not related to treatment but was related to outcome by ORBO. Outcome Oi was binary, representing dead (Oi = 1) or alive (Oi = 0). The new therapy was modeled to be effective with ORTO = 0.75; this small treatment effect was modeled so that confounding would bias results to produce an apparent detrimental effect in some individual-level models.
To model practice variability among hospitals, the probability of receiving treatment was dependent on the hospital. For most simulations, treatment was assigned using the following algorithm:
Logit [probability of treatment] = (0.04)H + [ln(ORCT)]C 0.9, (1)
where ln is the natural logarithm; this algorithm was chosen to ensure a large effect from confounding while allowing a wide range of treatment probabilities across the different institutions. Outcome was assigned to model the independent effects of the covariate B, the confounder C, and the treatment assignment, as follows:
Logit [probability of outcome] = [ln(ORBO)]B + [ln(ORCO)]C + [ln(ORTO)]T + 0.05. (2)
These equations ensure that there is no nonrandom association between the confounder and the grouped-treatment variable except through the actual treatment assignment.
In a previous analysis, we used generalized estimating equations to account for clustering of observations within institutions (9). To simplify these simulations and their interpretations, we assume here that there is no clustering of observations beyond that produced by treatment assignment. Therefore, logistic regression can be used instead of generalized estimating equations.
To test the reliability of the approach with different distributions of treatment probability among the hospitals, we generated additional cohorts: a uniform distribution, modeling a wide distribution of treatment frequency (probability of treatment = (0.01)H + 0.2 x C), and a split distribution, modeling frequent utilization of the treatment at only half the hospitals (probability of treatment determined by the uniform distribution, ranging from 0 percent to 10 percent at half the hospitals and from 35 percent to 60 percent at the other half). These distributions were chosen to reflect the type of variability in clinical practice that may be encountered in epidemiologic and health services research.
A total of 1,000 cohorts, corresponding to replicates, each with a sample size of 5,000, were generated according to a Bernoulli distribution for each set of conditions tested using the probabilities given by equations 1 and 2. An estimate of the treatment effect and its 95 percent confidence interval (by the Wald method) were calculated for each cohort using three different logistic models:
Model 1 (standard, individual-level analysis with known covariate and confounder):
Logit [probability of outcome] = ß1 T + ß2 B + ß3 C + K,
where ß1 represents the conditional treatment effect when there are no unmeasured confounders. K is a constant.
Model 2 (standard, individual-level analysis with known covariate and unmeasured confounder):
Logit [probability of outcome] = ß4 T + ß5 B + K,
where ß4 represents a confounded estimate of the conditional treatment effect.
Model 3 (unmeasured confounder and grouped-treatment variable (grouped-treatment estimator)). A grouped-treatment variable (GT) was determined as the proportion of patients treated by the new therapy at a given hospital:
Logit [probability of outcome] = ß6 GT + ß7 B + K,
where ß6 represents an unconfounded estimate of the treatment effect because there is no association between GT and outcome independent of treatment assignment T. The magnitude of ß6 is an estimate of the marginal treatment effectß4 from model 2 with ORCT set to 1because the condition in which all patients are treated by the new procedure (GT = 1) differs from the condition in which no patient is so treated (GT = 0) by the coefficient ß1. If the grouped-treatment analysis is unbiased, then ß6 should be equal to ß4 from model 2 with ORCT set to 1, which was estimated by simulation and is defined as the "true" value being estimated.
For each of the models, calculated coefficients from the logistic models listed above and the limits of the Wald 95 percent confidence interval were averaged for the 1,000 replicates. These values were converted to odds ratios. The confidence interval for each replicate was evaluated to test inclusion of the true underlying treatment effect: The calculated confidence interval would be expected to include the modeled marginal treatment effect (ß4 from model 2 with ORCT set to 1) in 95 percent of the replicates if it were determined accurately. Therefore, we tallied the proportion of replicates in which the calculated confidence interval included the actual marginal treatment effect, termed the percent coverage, to characterize the behavior of confidence intervals.
A number of variables were varied to determine the limitations of the grouped-treatment method as an approach to estimating the marginal treatment effect. Ranges of these variables were chosen to illustrate a variety of conditions that could occur in clinical practice and to test the method when confounding was severe enough to reverse the direction of the apparent treatment effect.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The accuracy of the grouped-treatment estimator (model 3) was tested over a range of association between the confounder and the treatment probability (ORCT) and compared with the standard, individual-level estimate with the confounder "unknown" and absent from the equation (model 2) (table 1). In these simulations, 5,000 patients and 50 hospitals were modeled, and all other associations were held constant. The underlying conditional treatment effect was small and protective (model 1; ORTO = 0.75). The prognosis was adversely affected by the confounder (ORCO = 2.7) and the covariate (ORBO = 2). Under these conditions, the underlying unconfounded marginal treatment effect is 0.76, the individual-level estimate (model 2) when ORCT = 1, and does not vary in the table. The association between the grouped-treatment variable and the actual treatment assignment was relatively weak in these cohorts (R = 0.260.29). The grouped-treatment estimates are generally closer to the unconfounded marginal treatment effect than are the standard individual-level estimates with the confounder unknown. Bias was present at the extremes, which were smaller than in the standard individual-level model and appeared to be in the direction of the underlying confounding. The confidence interval of the grouped-treatment estimates included the underlying marginal treatment effect in approximately 95 percent of replicates, as represented by the percent coverage, which suggests that the reliability of the estimate was accurately represented. This was in sharp contrast to the percent coverages for the individual-level estimator.
|
|
|
|
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
In our models, the grouped-treatment variable and the outcome are statistically independent conditional on the value of the actual treatment. This is equivalent to the basic instrumental variable assumption (10, 22). Practically, it corresponds with the assumption that the proportion of cases treated by a given procedure at an institution is not independently associated with the pretreatment prognosis of patients. In other words, high-risk patients are not being referred to certain hospitals because of those hospitals utilization of a specific procedure. For example, sicker patients with intracranial aneurysms do not preferentially seek treatment at those hospitals offering a new therapy. This assumption may be difficult to satisfy in practice and may limit the utility of the grouped-treatment approach.
If there are known confounders that are clustered at certain hospitals, such as ethnicity or older age, and these are also associated with outcome, including these variables in the model would be expected to eliminate the confounded association between the grouped-treatment variable and the outcome. For example, hospitals that treat wealthier patients may offer a new therapy more frequently. If wealth is associated with the outcome and there is no measure of wealth available, the grouped-treatment variable would be associated with outcome and the basic instrumental variable assumption would not be fulfilled. However, if an accurate measure for a patients wealth were included in the model, the assumption could be fulfilled. We found that adjustment for a covariate in the model with the grouped-treatment variable was possible, at least when the covariate was equally distributed among hospitals. Further analysis is required to confirm that adjustment for an unequally distributed covariate is practical and to demonstrate the impact of clustering of observations by institution due to unmeasured covariates that are not related to treatment assignment. In addition, adjustment for unequally distributed covariates that are measured only at the group level may be practical, but this requires further modeling and assumptions. Because unmeasured group-level covariates may be present, it may be advisable to use generalized estimating equations or random effects models to account for clustering (9).
As long as the basic instrumental variable assumption is fulfilled, the grouped-treatment estimator appears to provide a less biased assessment of the impact of treatment than does an estimator based on a standard individual-level model that does not include the confounder. Therefore, at a minimum the grouped-treatment estimator could provide a test of residual uncontrolled confounding in the standard analysis: A large difference between estimates based on the individual-level treatment variable and based on the grouped-treatment variable might suggest the presence of unmeasured confounding. A specific statistical test is available to compare results from standard and instrumental variable models (23), but a difference that is not statistically significant may still be important from a clinical or policy perspective, so a subjective comparison of the estimates may be more useful. In this respect, the grouped-treatment approach may have value as a sensitivity analysis.
The grouped-treatment approach is likely to provide more accurate estimates of the treatment effect if the correlation between the grouped-treatment variable and the actual treatment assignment is strong relative to the correlation between the grouped-treatment variable and the unmeasured confounder (24). The correlation between the grouped-treatment variable and actual treatment assignment increases with greater practice variability; in this instance, the likelihood of receiving treatment varies greatly from institution to institution, and this variation is independent of unmeasured confounders. As the correlation decreases, the grouped-treatment estimator is expected to become more biased in the direction of the standard, confounded estimator. Although we modeled three distributions of treatment utilization among the hospitals, the correlation between the actual treatment assignment and the grouped-treatment variable varied little (R = 0.190.28). The grouped-treatment approach may not perform as well when practice variability is very small.
It is possible to show that the grouped-treatment estimate is unbiased with large samples when there is no underlying treatment effectequivalent to the null hypothesis in many studies. Therefore, the approach should be reliable for hypothesis testing when the existence, rather than the magnitude, of a treatment effect is of primary interest. Additional theoretical research will be required to define the limits of the method and to develop more generalized models to accommodate nonnormal, continuous outcomes.
The confidence intervals for the grouped-treatment estimate were wider than those for the standard estimate. This was expected because of the loss of precision in estimating treatment by using the imperfect surrogate, the proportion of patients receiving the treatment at a given hospital. The confidence interval of the grouped-treatment estimate included the actual unconfounded marginal estimate in approximately 95 percent of replicates over a range of conditions. This was not the case for the individual-level model with the unmeasured confounder. Therefore, the grouped-treatment approach appears to produce confidence intervals with nominal coverage behavior.
In the example motivating these simulations and in our previous studies, we have defined the grouped-treatment variable as the proportion of cases receiving a certain procedure at the hospital of treatment. Other constructs for grouping the treatment are possible, such as geographic region or relative distance to a hospital with particular characteristics (12), and are valid as long as the grouped-treatment variable is independent of outcome, conditional on known covariates and the actual treatment assignment. Grouping based on the hospital of treatment has certain advantages. First, it is an intuitive unit so that the assumptions can be evaluated more directly. Second, the hospital of treatment is a convenient variable that is frequently available in large data sets. Third, practice variability between hospitals tends to exist whenever there is disagreement about the indications for a given treatment (5, 6), so it is relatively easy to demonstrate correlation between the grouped-treatment variable and actual treatment assignment. Fourth, if there are other hospital characteristics associated with the availability of a given treatment, such as greater hospital treatment volume, these can be evaluated in the models if measurable. For example, in our previous study of intracranial aneurysm treatment, we showed that the newer procedure was associated with lower risk of in-hospital death and that the effect was independent of hospital volume and offering other specialized procedures (8). If other hospital characteristics are not measurable, the results can still be interpreted at the hospital level: If hospitals that use the treatment more frequently have better outcomes, regardless of whether the treatment or other hospital characteristics are responsible, these hospitals may be good choices for referral of patients. However, the possibility that utilization of the treatment of interest may be associated with other group-level characteristics must be carefully considered. Such an association provides another potential source of unmeasured confounding.
Although the grouped-treatment approach produces biased estimates of the underlying marginal treatment effect, this bias appears to be smaller than that in standard models with only individual-level variables. In many instances, the bias in the grouped-treatment estimate appears quite small, and its confidence interval has good properties. Therefore, the grouped-treatment approach may supplement standard multivariable analyses of large data sets when the grouped-treatment variable has no independent association with the outcome because of unmeasured confounders and when the treatments being compared are generally options for the individuals included in the cohort.
![]() |
ACKNOWLEDGMENTS |
---|
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|