COMMENTARY

Bias in Intervention Studies That Enroll Patients From High-Risk Clinics

Sholom Wacholder

Correspondence to: Sholom Wacholder, PhD, Biostatistics Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD 20892-7244 (e-mail: wacholder{at}nih.gov)


    ABSTRACT
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
It is important to evaluate the effects of proposed interventions to reduce the risk of disease among carriers of a highly penetrant mutation, such as the mutations in BRCA1 and BRCA2 for breast and ovarian cancers or in APC and MLH1 or MSH2 for colon cancer. However, some studies that evaluate the effects of interventions designed to reduce risk in mutation carriers may be susceptible to a serious selection bias when they are based in clinics that care for persons at high risk for the disease. A study design in which a large fraction of the case patients were diagnosed before being seen at the clinic and all control subjects are persons previously seen at the clinic can create a false impression of intervention efficacy if, as is likely, mutation carriers seen at the clinic were more likely to receive the intervention than mutation carriers in the general population.



    INTRODUCTION
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
Effective interventions yield the greatest reduction in the burden of disease in populations at highest risk. Indeed, several interventions, including oral contraceptive use, prophylactic mastectomy, tubal ligation, and tamoxifen therapy, might reduce the high risks of breast and ovarian cancers among women who carry mutations in BRCA1 or BRCA2. However, evaluation of the efficacy of these interventions in persons who carry a highly penetrant mutation (e.g., APC and MLH1 or MLH2 mutations for colon cancer and women with BRCA1 or BRCA2 mutations for breast and ovarian cancers) is challenging for the following reasons. First, it can be difficult and expensive to assemble a study cohort that has enough mutation carriers to allow risk to be assessed. Second, individuals who carry highly penetrant mutations are not only rare but are typically ascertained only after they or someone in their family develops cancer. Even those population-based case–control studies in which participants with founder mutation alleles are relatively common typically have very few control subjects who are mutation carriers (13). Finally, the prospective information for evaluation of interventions (48) is often limited by the short follow-up time after the interventions were introduced.

Consequently, much of the published information on the efficacy of interventions intended to reduce the risk for disease comes from retrospective analyses of the medical records of individuals enrolled at high-risk clinics (i.e., clinics that counsel and treat patients who are at high risk for a particular disease). Retrospective studies of women from high-risk clinics that focus on BRCA1 and BRCA2 mutation carriers, who are at increased risk of breast and ovarian cancers, have evaluated the effects of potential interventions such as oophorectomy (911), mastectomy (12), tubal ligation (13), pregnancy (14), and the use of oral contraceptives (15,16) and tamoxifen (17). Here I describe a potential bias that may arise in studies that use populations from high-risk clinics to examine the effects of an intervention on risk.


    Advantages and Disadvantages of Clinic-Based Studies
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
Clinics that serve high-risk patients are a tempting source of individuals who carry high-risk mutations for use both in etiologic studies of the effects of cofactors (e.g., other genes, past and current behaviors, pregnancy, and the use of oral contraceptives) that could influence the risk of disease among carriers and in studies of interventions, such as prophylactic surgery, that are designed specifically to reduce risk of disease. Mutation carriers ascertained at high-risk clinics have agreed to and have received clinical genetic testing; therefore, their participation in a study does not entail added testing and counseling expenses. Furthermore, mutation carriers who have undergone clinical genetic testing are likely to be motivated to participate in studies of interventions to reduce their risk of cancer. However, a possible disadvantage of using mutation carriers from high-risk clinics in such studies is that they might be richer, better educated, more likely to be employed and have health insurance (at least in the United States) and, therefore, more receptive to medical interventions than mutation carriers not seen at a clinic. Thus, results from studies of mutation carriers identified and given care in a high-risk clinic may not be directly generalizable to mutation carriers identified and given care in a community setting.

Prospective follow-up of everyone seen at the clinic beginning at the time of first enrollment or when mutation carrier status is determined (48) is ideal. However, because a prospective study design may yield only small numbers of patients and short duration of follow-up, many retrospective clinic-based studies allow prevalent patients, i.e., those diagnosed with disease before being seen at the clinic, to be case patients.


    Bias
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
I now show that the retrospective approach can be problematic for factors that change over time, particularly when the cofactor is more likely to change in carriers seen at the clinic than in carriers in the general population. Specifically, clinic-based studies that examine the effect of an intervention in mutation carriers are susceptible to a serious, previously unrecognized bias when they include persons diagnosed with disease before being seen at the clinic as case patients but only include unaffected persons seen at the clinic as control subjects. Unaffected mutation carriers who are unaware of their elevated risks because they have not undergone clinical genetic testing and have not been seen previously at a high-risk clinic are less likely to receive an intervention for prophylaxis than identified mutation carriers who have enrolled at a high-risk clinic. As a result, estimates of the effect of the intervention will be overly optimistic. In particular, the case patients will have a lower frequency of an intervention that has no effect than the control subjects who are selected, by design, from the clinic.

The following hypothetical example shows how including case patients diagnosed before being seen at the clinic but requiring all control subjects to have been seen at the clinic could introduce bias. A BRCA1 or -2 mutation carrier diagnosed with breast cancer in 1990—the first member of her family to be affected—is included in a case–control study as a case patient. Clearly, this woman's family history gave her no reason to suspect that she was at increased risk, so she would not have undergone a surgical, medical, behavioral, or early detection intervention for prophylaxis before her breast cancer diagnosis. On the other hand, her sister, who was seen at the same clinic and identified as a mutation carrier in 1998, perhaps after yet another sister had been diagnosed with ovarian cancer, might have been urged to consider undergoing an intervention to reduce risk specifically because of her sisters’ histories and genetic test results. Thus, in the absence of an effect of the intervention, the case patients (e.g., women diagnosed with breast cancer in 1990) would be less likely to receive the intervention than potential control subjects (e.g., their unaffected sisters), and the intervention will appear protective. If the intervention does have a protective effect, the protection will be exaggerated; the negative effect of a deleterious intervention will be attenuated or possibly appear protective.

Table 1 presents a simple numerical example that illustrates how selection bias can create a false impression of benefit from an intervention in a clinic-based study. Consider a case–control study of the effect of an ineffective intervention received by 10% of the general population, regardless of their carrier status, family history of disease, or other determinants of risk. Furthermore, assume that half of carriers enrolling at the clinic without previous exposure to the intervention receive the intervention immediately upon enrollment and that the other half never receives the intervention. Then, 10% of 100 case patients diagnosed before going to the clinic and 55% [i.e., 10% + (50% x 90%)] of 100 case patients diagnosed after being seen at the clinic will have been previously exposed to the intervention at the time of diagnosis. Thus, 65 (32.5%) of 200 case patients will have been exposed to the intervention at the time of diagnosis. By contrast, 55% of the 200 control subjects drawn from clinic visitors will be exposed to the intervention. Thus, the odds ratio for the intervention would be 0.39 [i.e., (0.325/0.675)/(0.55/0.45)], suggesting protection, even though there was no benefit. Notice that in this scenario, there would be no selection bias if case patients and control subjects were restricted to individuals who were never seen at the clinic; the frequency of intervention would be 10% in each group. Similarly, there would be no selection bias if the case patients and control subjects were restricted to those who had not received the intervention before they enrolled in the clinic (because the frequency of intervention would be 50% for each group). Although only someone previously seen at the clinic can be selected as a control subject, a patient can be included as a case without having been seen at the clinic before diagnosis. Because having been seen at the clinic also is related to the probability of having received the intervention, epidemiologic theory says that this flawed study design is susceptible to selection bias (18).


View this table:
[in this window]
[in a new window]
 
Table 1. Hypothetical example of bias in a case–control study of an ineffective intervention among a population from a high-risk clinic*

 
The quantitative extent of the selection bias depends on two factors: 1) the difference between the fractions of persons that received intervention before and after being seen at the clinic and 2) the fraction of case patients in the study who were diagnosed before enrollment at the clinic. For example, the frequency of prophylactic interventions among mutation carriers who have been seen at high-risk clinics could be much greater than that observed among comparable mutation carriers who have not been seen at high-risk clinics. If, in addition, a sizable number of case patients were diagnosed before being seen at the clinic, the bias could be substantial.

The magnitude of the overestimate of benefit from the intervention in this constructed example is probably extreme. However, it is not usually possible to quantify the bias in an actual study without having the investigators provide information about how many cases were prevalent at enrollment at the clinic and the percentage of unaffected enrollees who receive the intervention at the clinic. There is, however, some information about both prospective-only and retrospective-only analyses in the recent study from the Prevention and Observation of Surgical Endpoints Study Group on the effects of prophylactic mastectomy on the risk of breast cancer (6). It is telling that the only two cases of breast cancer observed in women with a history of bilateral mastectomy, as well as 130 (87%) of the 149 breast cancer cases in women without mastectomy, needed to be excluded from the prospective analysis (referred to as "analysis 4") (6).


    General Principle
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
The key issue in clinic-based cohort or case–control studies of an intervention is whether receiving an intervention reduces risk of subsequent disease incidence. A randomized controlled clinical trial compares disease incidence after the participants are randomly assigned to receive the intervention or no intervention. In a nonrandomized clinic-based cohort study, either retrospective or prospective, follow-up time and case ascertainment appropriately begin when the individual is first seen at the clinic; that is, only diagnoses and person-time occurring after the first clinic visit should be included in the numerator and denominator used to calculate the incidence rate. Diagnoses and person-time occurring prior to the intervention are considered unexposed; diagnoses and person-time occurring after the intervention are considered exposed.

Because a case–control study is simply an efficient way to study the same cohort, a proper case–control study should adhere to the same eligibility restrictions and exposure definitions as a cohort study (18). If all control subjects were previously enrolled at the clinic, only those cohort members affected after being seen at the clinic should be included as case patients. When unaffected patients seen at the clinic are more likely to receive the intervention, the bias is more severe.

A retrospective approach can be adequate for cofactors that do not vary with time, such as germline DNA. But the value of the exposure defined by "history of intervention" can increase from 0 to 1, perhaps quite frequently immediately after an unaffected patient is first seen at the clinic. It is also important, of course, that the time of evaluation of case patients’ and control subjects’ exposures be comparable.


    Evaluating Effects of Interventions Received at a Clinic
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
Clinic-based studies will continue to be the most efficient way to identify mutation carriers for studying the effects of genetic and environmental factors that might modify risk and to assess the impact of interventions designed to reduce risk. How can studies that use subjects enrolled in high-risk clinics provide useful information about the effect of an intervention, yet avoid important bias? One option would be to restrict study eligibility of case patients to those who are diagnosed after their initial clinic visit to correspond with control subjects who are also unaffected persons when first seen at the same clinic. Alternatively, one could use only case patients diagnosed before coming to the clinic and consider the exposure history of mutation-carrying control subjects who were seen at the clinic only before they were first seen at the clinic; thus, only interventions that occurred in case patients and control subjects before they came to the clinic would be included in the analysis. Either strategy alone, or combining retrospective-specific and prospective-specific results, would ensure that the same dynamic criteria for membership in the cohort and inclusion for follow-up of persons seen at the high-risk clinic, and therefore for eligibility in the case–control study, would apply to both the case patients and the corresponding control subjects. Unfortunately, limitations with respect to statistical power considerations and practicality may hinder the application of each of these strategies (6).

Hartmann et al. (12) took a different approach in their study of the efficacy of bilateral prophylactic mastectomy in BRCA1 and BRCA2 gene mutation carriers. They did not try to monitor women who did not receive an oophorectomy for occurrence of cancer cases. Instead, they estimated the efficacy of the intervention by comparing the number of cases they observed in follow-up of women who received an oophorectomy to the number of cases expected in the absence of oophorectomy; the expected number of cases was calculated on penetrance estimates (1921). The validity of this method depends on how close the penetrance models are to the rates that would be observed in patients seen at the clinic had they not received oophorectomy.


    CONCLUSIONS
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 
Evaluating the effects of an intervention to reduce risk in mutation carriers at high risk of cancer by using patients from a high-risk clinic can lead to an overestimate of the protection or an underestimate of the harm associated with the intervention when many of the case patients are diagnosed before being seen at the clinic. The magnitude of the bias will be greatest for interventions rarely given outside high-risk clinics. For example, the estimate of risk reduction from oophorectomy is likely to be more distorted than the estimate of the effect of tubal ligation, which is more common. For interventions that are common among noncarriers (e.g., the use of oral contraceptives), the bias may be smaller but could still be of the same order of magnitude as the likely effect. This bias is unlikely to be important for a cofactor that does not change over time, such as germline mutations, or whose presence is not much influenced by whether an individual was seen at a clinic.

Several other biases may occur in studies of the effect of prophylactic surgery and other interventions in mutation carriers. Klaren et al. (22) identified, among others, "confounding by indication" (i.e., when individuals at greater risk are more likely to receive the intervention than individuals at lower risk) and "informative censoring" (e.g., censoring by death from ovarian cancer in a study whose endpoint is breast cancer). Additionally, use of prevalent cases can lead to a second bias. Patients with lengthy survival time after diagnosis will be overrepresented among case subjects. If the exposure affects survival time, the effect of exposure on risk of disease can be distorted.

There is not enough information available from published reports to quantify the bias at this point. It is likely, however, that bias is substantial in those clinic-based investigations in which a large fraction of case patients were diagnosed before being seen at the clinic and in which there is a large increase in frequency of the intervention among those seen at a clinic. Indeed, Rebbeck et al. (6) reported that a large fraction of the cases was identified retrospectively. It seems likely that personnel at high-risk clinics are quite persuasive at encouraging their already-predisposed patients to take interventions, so enrollment at the clinic is likely to affect the chance of receiving the intervention. Published reports of findings from such studies should include sufficient information to allow the reader to evaluate these potential biases.

In conclusion, when mutation carriers diagnosed before they have enrolled at a high-risk clinic are included as case patients and only unaffected mutation carriers seen at the clinic are included as control subjects, reports of interventions commonly given in high-risk clinics may have a substantial bias toward overestimating efficacy. This bias may render these studies too flawed to serve as the basis for patient management decisions in the decade or so before clear evidence from randomized trials of interventions to reduce the risk of disease is available. Although population-based cohort or case–control studies can provide some useful information, these studies are difficult to launch because mutation carriers are rare and difficult to identify, and analyses of these studies have limitations (23) even when feasible (1,2). For example, the discrepancy in the published data (1,15) about whether oral contraceptives exert a protective effect among BRCA1 and BRCA2 mutation carriers suggests that we may not yet have definitive answers to these important clinical questions. Investigators of clinic-based studies must either avoid including patients diagnosed before being seen at the clinic or show that the resulting bias has limited impact before clinic-based studies can reliably guide critical management decisions for persons at increased genetic risk for cancer.


    REFERENCES
 Top
 Abstract
 Introduction
 Advantages and Disadvantages of...
 Bias
 General Principle
 Evaluating Effects of...
 Conclusions
 References
 

1 Modan B, Hartge P, Hirsh-Yechezkel G, Chetrit A, Lubin F, Beller U, et al. Parity, oral contraceptives, and the risk of ovarian cancer among carriers and noncarriers of a BRCA1 or BRCA2 mutation. N Engl J Med 2001;345:235–40.[Abstract/Free Full Text]

2 Rutter JL, Wacholder S, Chetrit A, Menczer F, Ebbers S, Tucker MA, et al. Gynecologic surgeries and risk of ovarian cancer in women with BRCA1 and BRCA2 Ashkenazi founder mutations: an Israeli population-based case-control study. J Natl Cancer Inst 2003;95:1072–8.[Abstract/Free Full Text]

3 Modugno F, Moslehi R, Ness RB, Nelson DB, Belle S, Kant JA, et al. Reproductive factors and ovarian cancer risk in Jewish BRCA1 and BRCA2 mutation carriers (United States). Cancer Causes Control 2003;14:439–46.[CrossRef][ISI][Medline]

4 Struewing JP, Watson P, Easton DF, Ponder BA, Lynch HT, Tucker MA. Prophylactic oophorectomy in inherited breast/ovarian cancer families. J Natl Cancer Inst Monogr 1995;17:33–5.[Medline]

5 Kauff ND, Satagopan JM, Robson ME, Scheuer L, Hensley M, Hudis CA, et al. Risk-reducing salpingo-oophorectomy in women with a BRCA1 or BRCA2 mutation. N Engl J Med 2002;346:1609–15.[Abstract/Free Full Text]

6 Rebbeck TR, Friebel T, Lynch HT, Neuhausen SL, van 't Veer L, Garber JE, et al. Bilateral prophylactic mastectomy reduces breast cancer risk in BRCA1 and BRCA2 mutation carriers: the PROSE Study Group. J Clin Oncol 2004;22:1055–62.[Abstract/Free Full Text]

7 Olopade OI, Artioli G. Efficacy of risk-reducing salpingo-oophorectomy in women with BRCA-1 and BRCA-2 mutations. Breast J 2004;10 Suppl 1:S5–9.[CrossRef][Medline]

8 Meijers-Heijboer H, van Geel B, van Putten WL, Henzen-Logmans SC, Seynaeve C, Menke-Pluymers MB, et al. Breast cancer after prophylactic bilateral mastectomy in women with a BRCA1 or BRCA2 mutation. N Engl J Med 2001;345:159–64.[Abstract/Free Full Text]

9 Rebbeck TR, Levin AM, Eisen A, Snyder C, Watson P, Cannon-Albright L, et al. Breast cancer risk after bilateral prophylactic oophorectomy in BRCA1 mutation carriers. J Natl Cancer Inst 1999;91:1475–9.[Abstract/Free Full Text]

10 Rebbeck TR. Prophylactic oophorectomy in BRCA1 and BRCA2 mutation carriers. Eur J Cancer 2002;38 Suppl 6:S15–7.[Medline]

11 Rebbeck TR, Lynch HT, Neuhausen SL, Narod SA, Van't Veer L, Garber JE, et al. Prophylactic oophorectomy in carriers of BRCA1 or BRCA2 mutations. N Engl J Med 2002;346:1616–22.[Abstract/Free Full Text]

12 Hartmann LC, Sellers TA, Schaid DJ, Frank TS, Soderberg CL, Sitta DL, et al. Efficacy of bilateral prophylactic mastectomy in BRCA1 and BRCA2 gene mutation carriers. J Natl Cancer Inst 2001;93:1633–7.[Abstract/Free Full Text]

13 Narod SA, Sun P, Ghadirian P, Lynch H, Isaacs C, Garber J, et al. Tubal ligation and risk of ovarian cancer in carriers of BRCA1 or BRCA2 mutations: a case-control study. Lancet 2001;357:1467–70.[CrossRef][ISI][Medline]

14 Jernstrom H, Lerman C, Ghadirian P, Lynch HT, Weber B, Garber J, et al. Pregnancy and risk of early breast cancer in carriers of BRCA1 and BRCA2. Lancet 1999;354:1846–50.[CrossRef][ISI][Medline]

15 Narod SA, Risch H, Moslehi R, Dorum A, Neuhausen S, Olsson H, et al. Oral contraceptives and the risk of hereditary ovarian cancer. N Engl J Med 1998;339:424–8.[Abstract/Free Full Text]

16 Narod SA, Dube MP, Klijn J, Lubinski J, Lynch HT, Ghadirian P, et al. Oral contraceptives and the risk of breast cancer in BRCA1 and BRCA2 mutation carriers. J Natl Cancer Inst 2002;94:1773–9.[Abstract/Free Full Text]

17 Narod SA, Pal T, Graham T, Mitchell M, Fyles A. Tamoxifen and risk of endometrial cancer. Lancet 2001;357:65–6.[CrossRef]

18 Wacholder S, McLaughlin JK, Silverman DT, Mandel JS. Selection of controls in case-control studies. I. Principles. Am J Epidemiol 1992;135:1019–28.[Abstract]

19 The Breast Cancer Linkage Consortium. Cancer risks in BRCA2 mutation carriers. J Natl Cancer Inst 1999;91:1310–6.[Abstract/Free Full Text]

20 Easton DF, Ford D, Bishop DT. Breast and ovarian cancer incidence in BRCA1-mutation carriers. Breast Cancer Linkage Consortium. Am J Hum Genet 1995;56:265–71.[ISI][Medline]

21 Struewing JP, Hartge P, Wacholder S, Baker SM, Berlin M, McAdams M, et al. The risk of cancer associated with specific mutations of BRCA1 and BRCA2 among Ashkenazi Jews. N Engl J Med 1997;336:1401–8.[Abstract/Free Full Text]

22 Klaren HM, van't Veer LJ, van Leeuwen FE, Rookus MA. Potential for bias in studies on efficacy of prophylactic surgery for BRCA1 and BRCA2 mutation. J Natl Cancer Inst 2003;95:941–7.[Free Full Text]

23 Albert PS, Ratnasinghe D, Tangrea J, Wacholder S. Limitations of the case-only design for identifying gene-environment interactions. Am J Epidemiol 2001;154:687–93.[Abstract/Free Full Text]

Manuscript received November 6, 2003; revised June 9, 2004; accepted June 28, 2004.


This article has been cited by other articles in HighWire Press-hosted journals:


Related Memo to the Media

             
Copyright © 2004 Oxford University Press (unless otherwise stated)
Oxford University Press Privacy Policy and Legal Statement