Alternative treatments in reproductive medicine

The vexing problem of `seemingly impeccable trials....'

Jan P. Vandenbroucke

Department of Clinical Epidemiology, Leiden University Medical Center, C9-P, P.O.Box 9600, 2300 RC Leiden, The Netherlands. E-mail: j.p.vandenbroucke{at}lumc.nl


    Abstract
 Top
 Abstract
 Introduction
 References
 
The problem of `seemingly impeccable trials' that produce unbelievable results is approached from basic notions of the philosophy of science: facts and theory are interdependent, and `crucial experiments' do not exist. This does not lead to an `anything goes' attitude, but obliges us to consider arguments and counter-arguments in the spirit of the `crossword' analogy by Susan Haack. The role of editors and readers might be different: while editors might be under some obligation to publish `seemingly impeccable trials', readers are not obliged to accept the findings.

Key words: alternative treatment/epidemiological methods/evidence-based medicine/homeopathy/philosophy


    Introduction
 Top
 Abstract
 Introduction
 References
 
`The truth may be out there, but lies are inside your head' Terry Pratchett [Hogfather—a Discworld novel (1997) Corgi books, London, UK, p. 242].

The major concern that runs through the commentary by Dr Renckens (Renckens, 2002Go) is the problem of `seemingly impeccable trials proving absurd claims'. He is clearly vexed by it, and so am I. Let me start by introspection: why are we vexed? We are, because we are frustrated in our expectations. Our expectation is clear: if you conduct a `good experiment', this will show the truth. Because of that expectation, editors are under the moral obligation to publish `seemingly impeccable trials', whatever the results, and we seem forced to accept them. Yet, we clearly cannot.

Perhaps the problem is with our medical training, where there is no place for philosophy of science. If we were to put our problem to a professional philosopher, he would be much surprised, and most probably answer: `I could have told you so, it has nothing to do with alternative medicine; it is a universal phenomenon'. Philosophers of science know that `crucial experiments' that once and for all decide objectively what is true, do simply not exist. The very terms in which experiments are framed, the assumptions underlying the use of measurement instruments, as well as the interpretation of the measurements, are all based on theory. Facts and theories are much more indistinguishable than we like to think—neither one can exist without the other (Fleck, 1979Go; Vandenbroucke and de Craen, 2001Go).

At the most basic level the very production of facts is `loaded with theory'. That goes for history writing, as was so wonderfully described by Carr in his essay on the historian and his facts (Carr, 1964Go). It also goes for medical science: we will all delight in a subgroup analysis that confirms our expectations, but we will dismiss any subgroup analysis that seems nonsensical. Not only beauty is in the eye of the beholder, truth also is. Dr Renckens calls attention to the fact that the positive studies on homeopathy and the positive meta-analysis have been funded by organizations that defend (or produce) homeopathic products. The same happens in allopathic medicine. This very journal witnessed over the past 5 years the debate about `third generation oral contraceptives'. The epidemiologic studies that were sponsored by the pharmaceutical industry showed much lower relative risks than the unsponsored studies (Kemmeren et al., 2001Go). Even basic haemostatic tests turned out with different explanations in the hands of different authors.

Does all of this mean that `anything goes', i.e. that science can be replaced by poetry, since all facts are facts only because of their interpretation, and we can all have different interpretations? It does not. The best solution that I know of has been described by the philosopher Susan Haack in the crossword analogy which she explains for lay audiences (like medical doctors) as follows: `The clues [of the crossword] are the analogue of experiential evidence, already-completed entries the analogue of background information. How reasonable an entry in a crossword is depends upon how well it is supported by the clue and any other already intersecting entries; how reasonable, independently of the entry in question, those other entries are; and how much of the crossword has been completed. An empirical proposition is more or less warranted depending on how well it is supported by experiential evidence and background beliefs; how secure the relevant background beliefs are, independently of the proposition in question; and how much of the relevant evidence the evidence includes. How well evidence supports a proposition depends on how much the addition of the proposition in question improves its explanatory integration. There is such a thing as supportive-but-less-than-conclusive evidence, even if there is no formalizable inductive logic' (Haack, 1998Go).

The gist of the quote is that in discussing `facts', we should not only discuss the experiments that produced them, but also the plausibility of the theory behind it. A rather tentative long entry in a crossword will be quickly rejected if several plausible short entries do not immediately fit. However, when a long entry is very secure, given all the other long and short entries, for the next entry we will search and search until we find a word to make it fit. As I have argued more extensively elsewhere (Vandenbroucke and de Craen, 2001Go) this goes for allopathic as well as for homeopathic medicine.

Homeopathy remains a good example. We do not accept the crossword entry that infinite dilutions work since that cannot fit with existing physics and chemistry. When we are confronted with seemingly impeccable trials that `prove' that homeopathy works, we take a hard look at this evidence. So much was shown by Sterne et al. when performing a meta-regression on the homeopathic trial results (Sterne et al., 2001Go). From that analysis it followed that when trials were unblinded or small, they produced a larger effect of homeopathy. The more the trials were `large and blinded', the smaller the effect. The authors were quick to point out that their analysis cannot prove that homeopathy does not work. Nevertheless, it makes the crossword fitting, and because it does so, we gladly accept these meta-regression results.

The hard lesson is that there exists no single foolproof method to distinguish truth from absurdity. After all, the odd observation that strikes us—the time honoured `case reports' in medicine—can lead to a new line of discovery (Vandenbroucke, 2001Go). The first report that prone sleeping position was associated with sudden infant death syndrome was met by disbelief. There was `no mechanism', and even some physiologic reasoning to the contrary. However, the finding was put to test, to arguments and counter-arguments. Potential bias and confounding was searched for, studies replicated, and after a long itinerary the findings were accepted. In the end it is this type of critical discussion where we stick to arguments and counter-arguments that gives us the thorny path to truth. However, such discussion has a certain `ethos'. It does not suffice to say that `bias and confounding' can explain the result of a study. One should specify the potential bias, so that it can be put to test. After it has been put to test, either by reanalysis of data, by the collection of new data or by logical reasoning, the objection should be either accepted or laid to rest. Perhaps this means that editors still have the obligation to publish `seemingly impeccable trials' for the sake of discussion—but that we readers are under no obligation to accept these results.

In scientific discussion, we should resist the tendency to `shoot from the hip'. If some observation has no theoretical basis at all, we should be on our guard. Nevertheless, the example of case reports about side-effects shows that they are more often right than wrong (Vandenbroucke, 2001Go)—so, the astute physician can pick up something, even without theoretical basis. If an observation goes against accepted theory, we still have the obligation to think and articulate very clearly why we dismiss `the fact', that is why we dismiss the theory behind it. During a lecture that I once held about these topics, I could tell from the body language of the audience that many felt uneasy by my derogatory remarks about infinite dilutions in homeopathy. However, when I told them that blinded trials about distant prayer had come out positive, they burst out in laughter. Such is the difference in credulity of a 21st century audience.


    References
 Top
 Abstract
 Introduction
 References
 
Carr, E.H. (1964) What is History? Penguin, London, UK

Fleck, L. (1979) Genesis and Development of a Scientific Fact [Transl]. University of Chicago Press, Chicago, USA.

Haack, S. (1998) Manifesto of a Passionate Moderate. University of Chicago Press, Chicago, USA.

Kemmeren, J.M., Algra, A. and Grobbee, D.E. (2001) Third generation oral contraceptives and risk of venous thrombosis: meta-analysis. Br. Med. J., 323, 131–134.[Abstract/Free Full Text]

Renckens, C.N.M. (2002) Alternative treatments in reproductive medicine: much ado about nothing. Hum. Reprod., 17, 528–533.[Abstract/Free Full Text]

Sterne, J.A.C., Egger, M. and Smith, G.D. (2001) Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. Br. Med. J., 323, 101–105.[Free Full Text]

Vandenbroucke, J.P. (2001) In defense of case reports and case series. Ann. Intern. Med., 134, 330–334.[Abstract/Free Full Text]

Vandenbroucke, J.P. and de Craen, A.J. (2001) Alternative medicine: a `mirror image' for scientific reasoning in conventional medicine. Ann. Intern. Med., 135, 507–513.[Abstract/Free Full Text]





This Article
Abstract
FREE Full Text (PDF )
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Request Permissions
Google Scholar
Articles by Vandenbroucke, J. P.
PubMed
PubMed Citation
Articles by Vandenbroucke, J. P.