Department of Clinical Alcohol Research, Lund University, Malmö University Hospital, Malmö and
1 The Swedish Council on Technology Assessment in Health Care, Stockholm, Sweden
(Received 1 April 2003; accepted 1 April 2003)
We appreciate the careful reading and evaluation of the Swedish Council on Technology Assessment in Health Care (SBU, 2001) report by Dr Poikolainen (SBU, 2002). Poikolainen discusses several important issues, not only related to the report but also to systematic reviews in general. First, we shall comment on the SBU methodology.
SBU METHODOLOGY
It is important to stress that the methodology applied is basically a series of systematic reviews of the available literature. The use of meta-analytical procedures depends on the character of the studies on the different topics. In most previous SBU reports meta-analytical techniques have not been applied. In our opinion, the development of meta-analytical methodology and the availability of effective software for performing meta-analytical calculations have made this technique simple and more applicable for systematic literature reviews. The number of meta-analytical reviews in most scientific fields, including alcohol and drug treatments, have also rapidly increased in recent years.
It is also important to stress that the procedure after the completion of the reviews, including a first draft of Councils executive conclusions, and leading to the final report, is rather extensive. It includes reviews by external experts and by the SBUs Executive Board and Scientific Advisory Board, where the evidence was carefully evaluated. The process from final review to publication took more than a year.
We should now like to acknowledge some shortcomings of the Swedish version of the SBU report and then to discuss a few of the issues raised by Poikolainen.
SHORTCOMINGS OF THE SWEDISH VERSION OF THE SBU REPORT
Incorrect presentation of effect sizes
Unfortunately, in some of the meta-analyses in the chapter on psychosocial treatments and pharmacological treatments of alcohol dependence, and pharmacological treatments of drug dependence, log odds, instead of d with Hedges correction, have been presented as the effect size. This increases the effect size, but does not influence levels of significance. This has been corrected in the English version, and a correction will also be made available in Swedish.
Meta-analytical procedures did not include calculations of homogeneity/heterogeneity and search for moderator effects
In the English version we have recalculated most of the meta-analyses using the software program of Borenstein and Rothstein (1998). Tests for heterogeneity have been performed and, if positive, a search for moderator effects has been done. If no moderator has been identified, the results both of a fixed and of a random model are presented. With these changes we regard the meta-analyses being up-to-date concerning methodology.
In his commentary, Poikolainen states as if it were a matter of fact that only studies with similar patient groups should be included in a meta-analysis. However, this is open to debate. Obviously, studies must have some common elements if it is to be meaningful to pool them. If, on the other hand, studies are too similar, one loses the opportunity for generalizability that Poikolainen demands. So using a random-effects model, when statistical heterogeneity is detected or, in many cases, even when the test for heterogeneity is not significant, it is possible to pool rather different studies. If level of severity, dose, intensity, treatment duration or other moderator variables do not make too great a difference, generalizability is present. We could have further developed this type of analysis if time had permitted. Search for moderator effects using meta-regression is, however, a retrospective procedure with considerable risk for spurious relationships, especially when few studies are available in the meta-analysis.
Lack of chapter on methodology
It would be an advantage to have a general chapter on methodology. In the Swedish version methodological aspects are presented in the individual chapters. The main outcome measures have been related to misuse: in alcohol dependence the abstinence rate and in some cases the rate of not returning to heavy drinking, and in drug dependence both misuse and retention. Other outcome measures have generally not been included in the analyses, with the exception of the studies on psychiatric comorbidity in which psychiatric outcome methods have been included. A major reason for not using more functional outcomes is the fact that most primary studies do not measure them. In the English publication, a general methodology chapter has been added.
Problems with data abstracting
We acknowledge problems with data abstracting. We have carefully reviewed most of the critical studies again and corrected obvious faults. Many decisions in the data abstracting, however, are not clear-cut. A statistician has been consulted in several of the more complicated cases. Data abstracting has been considerably improved in the English version. However, the abstracting procedure is very costly and time-consuming and could be further improved (e.g. by contacting the authors of the papers). The improvements have so far not influenced the main conclusions.
ADDITIONAL ISSUES
D-statistics and dichotomous data
We acknowledge problems with d-statistics and dichotomous data, which have been discussed extensively in the literature, as Poikolainen points out. It is, however, an acceptable procedure from a statistical point of view. Effects will always be influenced by the baseline risk (result in the control group), regardless of the outcome being analysed dichotomously or continuously.
This methodology has been successfully used by Moyer et al.(2002) concerning secondary prevention and psychosocial treatment in alcohol problems. The success of their attempt was acknowledged by the editorial comments on the paper (Heather, 2002
). They used a similar approach as the one we use in the English publication. Most of their analyses were homogenous from a statistical point of view.
Brief intervention
With the publication of the excellent meta-analysis by Moyer et al. (2002) in Addiction, the lack of a meta-analysis in the SBU report can certainly be criticised. However, the qualitative conclusion is still valid. The suggested contradiction in the executive summary is mainly caused by careless reading. The summary states that we have no strong scientific support for describing a certain level of alcohol intake as safe. In practice, the summary continues, researchers are in reasonable agreement. It is this somewhat pragmatic agreement that makes the foundation for the calculations of the number needed to treat (NNT).
Project MATCH study
It is correct that in the Project MATCH Research Group (1997) study, the authors conclude that there are no clinical significant differences between the different therapies described in the article. However, in the same article they report some significant differences between the therapies according to the primary efficacy variables as well as in some of the secondary variables.
Review verdict
As it is the scientific community and not the prosecutor, Poikolainen, that passes the verdict, we rest assured that our main conclusions are strongly supported by the studies reviewed and that, although Poikolainen has made several helpful comments, the statement that the main findings are not to be trusted is unfounded and not supported by the review presented.
CONCLUSIONS
In conclusion, the ambitions of the SBU report, to include the entire literature concerning treatment of alcohol and drug dependence, has some limitations concerning precision and sophistication in methodology compared with smaller reviews and analyses. However, using similar techniques in data abstracting with one statistician advising all authors, comparability between the different chapters and their conclusions probably could be assumed to be better than in a total body of smaller independent analyses. We acknowledge several weaknesses in the Swedish report, which we have corrected in the English version. No changes in analysis and data abstractions, however, have had any influence on the initial findings.
FOOTNOTES
* Author to whom the correspondence should be addressed at: Department of Clinical Alcohol Research, Malmö University Hospital, Entr. 108, S-205 02 Malmö, Sweden. Email: mats.berglund@alk.mas.lu.se
REFERENCES
Borenstein, M. and Rothstein, H. (1998) Comprehensive Meta-analysis. A Computer Program for Systematic Reviews. Biostat, Englewood.
Heather, N. (2002) Effectiveness of brief intervention proved beyond reasonable doubt. Addiction 97, 293294.[CrossRef][ISI][Medline]
Moyer, A., Finnery, J. W., Sveringen, C. E. and Vergum, P. (2002) Brief intervention for alcohol problems: A meta-analytic review of controlled investigations in treatment-seeking and non-treatment-seeking populations. Addiction 97, 279292.[CrossRef][ISI][Medline]
Poikolainen, K. (2002) Invited Commentary: A nice try that fails: The Swedish Council on Technology Assessment in Health Care (SBU) evaluation of the effect of treatment of alcohol and drug problems: The epidemiologists view. Alcohol and Alcoholism 37, 416418.
Project MATCH Research Group (1997) Matching alcoholism treatments to client heterogeneity: Project MATCH post-treatment drinking outcomes. Journal of Studies on Alcohol 58, 929.
SBU (2001) Behandling av alcohol-och narkotikaproblem. En evidensbaserad kunskapssamman ställning. Staten beredning för medicinsk utvärdering, rapport nr 156. Stockholm. ISBN: 91-87890-73-9. Available online at http://www.sb.se/admin/index.asp