Department of Clinical Epidemioloy, Leiden University Medical Centre, Bldg 1 C9-P, 2300 RC Leiden, The Netherlands. E-mail: J.P.Vandenbroucke{at}lumc.nl
In 1995, a long time before any result of any randomized controlled trial would be known, I wrote that about half of the alleged 3545% reduction of myocardial infarction with hormone replacement therapy (HRT) in observational studies would not be real.1 Still, I should confess that the total reversal of the effecta slight increase of myocardial infarction in the first years of usetook me also by surprise.
Why was I so certain that a large part of the seeming protection from myocardial infarction would not be trueeven in the face of an overwhelming number of epidemiological studies?2 I guess, in essence, because during my training in epidemiology at the Harvard School of Public Health in 1978/79, I was much influenced by Miettinen's argument about the difference between studying the intended and unintended effects of treatments.3 The view of the believers of the HRT protection was summarized in a lecture by a leading gynaecologist that I attended in the 1990s: he showed beautiful coloured slides about the protective effect of HRT, telling his audience that all possibility of bias and confounding was ruled out by careful statistical adjustments in well-designed studies. He also had one slide in black and white, showing the possibility of an increase of venous thrombosis with HRT, but added that this would most certainly be due to bias. Actually, the theory on the difference between studying intended and unintended effects predicts exactly the opposite. Adverse effects are unintended, therefore generally unexpected and unpredictableat least in the usual clinical consultation. Thus, there is little likelihood of bias and confounding for adverse effects of HRT, such as venous thrombosis. One can even increase the unpredictability of adverse effects by limiting a study to people without any risk factor for the adverse effect, as was described already in 1978 by Jick and Vessey4for example, by limiting studies on venous thrombosis and different oral contraceptives to otherwise healthy young women without any risk factor for thrombosis.
In contrast, signs were all over the wall that prescription of HRT was highly selective against risk factors of myocardial infarction. Physicians who prescribed HRT in the 1970s and 1980s were risk avoiding about coronary risk factors, as was described by many authors (see ref. 1 for overview of critical opinions from the time before the RCT on HRT). This risk averse prescribing behaviour has persisted until the year 2000.5 Coronary disease has many clinically recognisable risk factors, such as hypertension, hypercholesterolaemia, and diabetes. If prescription is with an eye on such risk factors, and in particular if many complex and interdepedent risk factors come into play, the confounding bias introduced by this prescription process cannot be grasped any more by simply adjusting for a few variables in a logistic regression.3 A similar argument was used by Rubin in defence of randomization: he explained that a clinician's treatment decisions are just too complex to be modelled, and that therefore randomization is always preferable.6
In later works, Rubin and Rosenbaum coined the concept of strong ignorability of the allocation mechanism: in specific subgroups, or in specific circumstances, the allocation mechanism might be ignored: it can be left out of a model when it plays no role in determining the outcome.7 In general, adverse effects will meet that condition because they are most of the time unpredictable, or not taken into account, when prescribing.3,4 Thus, for adverse effects observational studies will be as good as randomized trials. Indeed, for the adverse effects of HRT, such as venous thrombosis and breast cancer, observational and randomized studies concurred almost on the dot, at least in terms of relative risk.8,9
The main lesson of this episode to me is a vindication of the old theories on intended and unintended effects and the investigation of adverse drug reactions.3,4 To study the effect of a treatment that is prescribed on the basis of complex rules of indication and contraindications, observational studies run a high risk of getting the answer wrong. If there is no form of sufficiently haphazard allocation of the exposure at baseline, it will not help to throw a few variables into a model for adjustment, and then hope that the causal risk has been quantified.10 In observational research, such haphazard allocationsi.e. allocations that are not intrinsically tied to prognosisare most likely for unexpected and undesired effects. Haphazard allocation is not the same as physical randomization. However, as stated by Rosenbaum:
Haphazard is not random ...I have generalized these arguments in a separate paper proposing a three-pronged restriction to give observational studies the best chances to be as credible as randomized trials.12 In thinking about the evaluation of quality of research, the distinction between intended and unintended effectsrather than a mere hierarchy with the randomized trial on topmight also be the way forward.13Still, haphazard or ostensibly irrelevant assignments are to be preferred to assignments which are known to be biased in ways that cannot be measured and removed analytically.11
![]() |
References |
---|
![]() ![]() |
---|
2 Stampfer MJ, Colditz GA. Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. Prev Med 1991;20:4763. (Reprinted Int J Epidemiol 2004;33:44553.)[ISI][Medline]
3 Miettinen OS. The need for randomization in the study of intended effects. Stat Med 1983;2:26771.[Medline]
4 Jick H, Vessey MP. Case-control studies of drug induced illness. Am J Epidemiol 1978;107:17.[ISI][Medline]
5 The Million Women Study. Patterns of use of hormone replacement therapy in one million women in Britain, 19962000. Br J Obstet Gynaecol 2002;109:131930.
6 Rubin DB. Bayesian inference for causal effects: the role of randomization. Ann Stat 1978;6:3458.[ISI]
7 Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983;70:4155.[ISI]
8 Beral V, Banks E, Reeves G. Evidence from randomised trials on the long-term effects of hormone replacement therapy. Lancet 2002;360:94244.[CrossRef][ISI][Medline]
9 Grodstein F, Clarkson TB, Manson JE. Understanding the divergent data on postmenopausal hormone therapy. N Engl J Med 2003;348:64550.
10 Goldthorpe JH. Causation, statistics and sociology. Eur Sociol Rev 2001;17:120.
11 Rosenbaum PR. Observational studies 2nd Edn. New York: Springer, 2002, p. 345.
12 Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet 2004 (In press).
13 Glasziou P, Vandenbroucke JP, Chalmers I. Assessing the quality of research. BMJ 2004:328:3941.