Department of Epidemiology, University of Maastricht, The Netherlands.
SirsWe thank Professors Bracken1 and Savitz2 for their positive and stimulating commentaries on our article on false positive and true positive outcomes in occupational cancer epidemiology.3 Bracken rightly puts the finger on one of the key complicating elements of our analysis, the lack of a true gold standard. Indeed as Bracken phrases it: our study of false positive and true positive outcomes in itself may very well be a false positive study. The lack of a true gold standard implies that our distinction between false positive and true positive outcomes contains some degree, hopefully only marginal, of subjectivity.
Our analysis was prompted by two considerations. First, epidemiology is a discipline with great intrinsic interest in methodological issues. In general, a well-trained epidemiologist will never rule out the possibility that the outcome of a study is influenced by the applied research methods. There is, and should be, always that voice in the back of the epidemiologist's mind that some research outcome may be an outlier, a chance finding or indeed an artefact caused by research methodology and conventional research strategies. This sense of intrinsic doubt regarding research outcomes is not always shared by people outside the scientific community. One of the areas in which epidemiological studies are reviewed and interpreted by non-epidemiologists is the arena of risk assessment and standard setting for the occupational environment. Because of the more experimental approach taken by toxicologists they tend to take research findings at face value and for instance have great difficulty in combining the results of conflicting outcomes of epidemiological studies. An example: cohort studies of butadiene exposed workers show an increased risk for leukaemia in production workers and an excess risk for lymphoma in processing workers.4 An epidemiologist would be inclined to see these two results as inconsistent and perhaps even conflicting, since no lymphoma risk was observed in production facilities and no leukaemia risk was observed in processing plants. However, scientists from other disciplines can be inclined to interpret these results differently, namely as evidence for the existence of a leukaemia risk as well as a lymphoma risk. It is not always easy for non-epidemiologists to put the results of epidemiological studies into their proper perspective, certainly if the results are inconsistent and contradictory.
The second consideration in performing our literature review and analyses was that there is always great debate among epidemiologists about possible biases and design shortcomings in epidemiological studies that supposedly have biased the outcomes. Most of these debates are not based on facts or evidence. Given the intensity with which evidence based medicine is advocated in clinical epidemiology it is surprising to see how weak the evidence is for most of the research methods in epidemiology. Most methods are based on convention and sound logic and the notion that epidemiology works well in most cases, but not on solid evidence.
We agree with the comments made by Savitz2 that a strong distinction between positive and negative study outcomes should perhaps not be made as rigorously as was done in our analysis. True, a study will add a variable degree of evidence in the form of confirming or refuting a specific hypothesis. In our opinion the creation of the Hypothesis Generating Machine by Cole5 has not altered the necessity of having a formal a priori hypothesis that will be tested. Merely referring to the Hypothesis Generating Machine can never be an appropriate replacement for a well-defined specific hypothesis to be tested. True, basing one's research on a specific hypothesis to be tested does not alter nature and consequently should not affect the validity of the outcome. However, we observed a strong tendency for studies that lack a formal specific hypothesis to be more likely to be false positive. This finding is in clear contradiction with the underlying reasoning of the Hypothesis Generating Machine installed by Cole and we propose to unplug it and put it in the scrap yard. Our study indicates that having a well-defined and specific a priori hypothesis is a valuable asset for any epidemiological study and we consequently propose to give more weight of evidence to studies that have been based on a specific and well-defined hypothesis.
In analogy with Cole's Hypothesis Generating Machine, fishing expeditions may be used by scientists as a sort of Publication Generating Machine, which will most likely produce some kind of positive outcome, worthy of publication. Of course these Publication Generating Machines should also be sent to the scrap yard. Because of the substantial pressure to publish in academia one would expect that relatively more of these Publication Generating Machines are installed in universities. Our study did not find evidence for this, since only 38% of the studies from universities were classified as fishing expeditions compared to 49% in the other affiliations combined. These findings of course have their limitations too. It is very easy to come up with an alternative explanation. Perhaps researchers at universities are more familiar with the debates about Bayesian theory and formal hypothesis testing and are more inclined to disguise their fishing expedition by adding some fancy hypothesis in the introduction.
In summary, we attempted to contribute to the further development of epidemiological methods by observing the nature of epidemiological practice in a systematic manner. To our great surprise case-control studies were just as likely to yield false positive results as cohort studies. What really helps in avoiding false positive outcomes is to have a formal a priori hypothesis, to adjust for smoking and other confounders and to measure or quantitatively assess the relevant exposure under investigation.
Finally, it is noteworthy to observe the great efforts in current research to develop and expand a scientific basis for medicine, evidence based medicine, but epidemiologists should not forget the need for a scientific basis for their own discipline. Thus we advocate epidemiology should invest in Evidence Based Epidemiology.
Notes
a Published in Int J Epidemiol 2001;30(5).
References
1
Bracken MB. Commentary: Toward systematic reviews in epidemiology. Int J Epidemiol 2001;30:95457.
2
Savitz DA. Commentary: Prior specification of hypotheses: cause or just a correlate of informative studies? Int J Epidemiol 2001; 30:95458.
3
Swaen GMH, Teggeler O, van Amelsvoort LGPM. False positive outcomes and design characteristics in occupational cancer epidemiology studies. Int J Epidemiol 2001;30:94854.
4 Himmelstein MW, Acquavella JF, Recio L, Medinsky MA, Bond JA. Toxicology and epidemiology of 1,3-Butadiene. Crit Rev Toxicol 1997; 27:1108.[ISI][Medline]
5 Cole P. The hypothesis generating machine. Epidemiology 1993;4: 27173.[ISI][Medline]