Pitfalls in conducting systematic reviews of acupuncture

A. White and E. Ernst

Peninsula Medical School, Universities of Exeter and Plymouth, Exeter, UK

Correspondence to: A. White, Complementary Medicine, Institute of Health and Social Care Research, Peninsula Medical School, 25 Victoria Park Road, Exeter EX2 4NT, UK. E-mail: Adrian.White{at}pms.ac.uk

SIR, A recent article explored some of the problems in acupuncture research, particularly how such research can mislead an ‘acupuncture-naïve reader’ [1]. The authors base their article very much on our early review of acupuncture for neck pain [2]. They are to be congratulated for highlighting some of the limitations of reviews. However, their article itself inadvertently illustrates some of their inherent difficulties.

First, basic terminology: even ‘acupuncture-naïve readers’ may be likely to understand the difference between evidence of effectiveness (i.e. in usual practice, compared with another form of treatment or no treat-ment) and evidence of efficacy (i.e. in ideal conditions compared with placebo). They might therefore have been left con-fused by the phrase which was used in the article as a broad conclusion of our review: ‘Acupuncture treatment for neck pain is not supported by evidence’. Our review’s bottom-line conclusion was about efficacy: but we included in the text evidence of acupuncture’s effectiveness compared with waiting-list and possibly against physiotherapy. All systematic reviews deserve to be read carefully; sound-bite summaries are often misleading.

White and colleagues take us to task in some detail for the inclusion criteria we applied. In doing so, they state their own opinion as if it was fact. The authors state that trials of laser acupuncture ‘should have been excluded’ from the review because, in their opinion, laser acupuncture is not acupuncture. While etymology may be on their side (and initially we planned to exclude trials of laser for that very purist reason), our hypothetical naïve consumer is likely to have a more pragmatic viewpoint. All three studies of ‘laser acupuncture’ were actually described as ‘acupuncture’, or used the laser at acupuncture points. Laser acupuncture is often presented to naïve consumers as ‘acupuncture’, and they will want to know whether it is effective. The consumer’s interest therefore indicated that we should set etymological purity aside and include trials of laser acupuncture. It is well to remember the words of another reviewer of reviews: ‘Often there is no right or wrong answer in what should be included’ [3].

The next point is more technical: the authors state that evidence gathered must have external validity. Reviewers actually have little control over this desirable feature, as the external validity of a review is rather dependent on that of the primary studies. But we were accused of a ‘misrepresentation of the truth’ for not taking into account certain factors, such as whether the acupuncture treatment was pragmatic and generalizable. We agree that it is important for primary researchers to make sure their studies are generalizable, for example by using standard treatments and meaningful controls. The authors have themselves conducted primary research in neck pain, and regrettably they have not followed their own advice. They either used a very rare form of acupuncture with something called ‘IP cords’ [4], or they compared acupuncture with sham TENS (transcutaneous electrical nerve stimulation) [5], a control which is considered controversial in acupuncture research methodology [6] because the placebo effect of sham TENS is likely to be very different from that of sham-acupuncture.

The authors of the article [1] also highlight what they see as straightforward errors in our review. On closer checking, we find it is they who have made the errors. Our Jadad scores were correct (we can willingly show them the details); we are easily able to calculate the total number of treatments correctly when three or four treatments a week (stated in the report) are given for 12 weeks (stated in the abstract!). They also suggest that we should have reported the results of short-term outcomes separately: we agree, and we did.

We join with the authors in their call for ‘new and more appropriate methodology’ and have ourselves already contributed to the debate [7]. In addition to the checklist offered by White et al. [1], we also considered the effects of database and language restrictions, and trial size. The authors might also have quoted other advances in the area of performing [8] and reporting acupuncture trials [9].

The reader should not be distracted from this article’s main message by its shortcomings. Yet the reader is likely to be confused again in the last section, on seeing the statement that ‘the only way forward ... is not to re-examine existing trials in systematic reviews’ because of the trials’ poor quality and homogeneity. But this poor quality and homogeneity can best, or perhaps only, be demonstrated by performing systematic reviews. And it is only by re-examining the trials, along with any new evidence, that we provide our patients with the best summary of the evidence, a policy at the centre of the Cochrane Collaboration.

The authors have declared no conflicts of interest

References

  1. White P, Lewith G, Berman B, Birch S. Reviews of acupuncture for chronic neck pain: pitfalls in conducting systematic reviews. Rheumatology 2002;41:1224–31[Abstract/Free Full Text]
  2. White AR, Ernst E. A systematic review of randomized controlled trials of acupuncture for neck pain. Br J Rheumatol 1999;38:143–7[CrossRef]
  3. Linde K, Willich SN. How objective are systematic reviews? Differences between reviews on complementary medicine. J R Soc Med 2003;96:17–22[Abstract/Free Full Text]
  4. Birch S, Jamison RN. Controlled trial of Japanese acupuncture for chronic myofascial neck pain: assessment of specific and nonspecific effects of treatment. Clin J Pain 1998;14:248–55[CrossRef][ISI][Medline]
  5. White P. The efficacy of acupuncture for the treatment of chronic mechanical neck pain – a randomised, controlled, single blind trial. Complement Ther Med 2002;10:113
  6. Ter Riet G, Kleijnen J, Knipschild P. Acupuncture and chronic pain: a criteria-based meta-analysis. J Clin Epidemiol 1990;43:1191–9[ISI][Medline]
  7. White AR, Trinh K, Hammerschlag R. Performing systematic reviews of clinical trials of acupuncture: problems and solutions. Clin Acupunct Orient Med 2002;3:26–31[CrossRef]
  8. White AR, Filshie J, Cummings M. Clinical trials of acupuncture: consensus recommendations for optimal treatment, sham controls and blinding. Complement Ther Med 2001;9:237–45[CrossRef][ISI][Medline]
  9. MacPherson H, White A, Cummings M, Jobst K, Rose K, Niemtzow R. Standards for reporting interventions in controlled trials of acupuncture: The STRICTA recommendations. STandards for Reporting Interventions in Controlled Trials of Acupuncture. Acupunct Med 2002;20:22–5 (also at http://www.medical-acupuncture.co.uk/journal/trials.shtml)[Medline]
Accepted 25 February 2003





This Article
Full Text (PDF)
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Disclaimer
Request Permissions
Google Scholar
Articles by White, A.
Articles by Ernst, E.
PubMed
PubMed Citation
Articles by White, A.
Articles by Ernst, E.
Related Collections
Other Rheumatology