1 Health Services Research Unit, University of Aberdeen, Aberdeen; 2 Department of Microbiology, Frenchay Hospital, Bristol; 3 Department of Pathology and Microbiology, University of Bristol, Bristol; 4 MEMO, Department of Clinical Pharmacology, University of Dundee, Dundee DD1 9SY, UK
Received 18 June 2003; returned 17 August 2003; revised 28 August 2003; accepted 28 August 2003
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods: We have systematically reviewed the literature from 1980 to identify interventions that alone, or in combination, are effective in improving antibiotic prescribing to hospital inpatients. The protocol was peer reviewed and has been published by the Effective Practice and Organization of Care (EPOC) Group of the Cochrane Collaboration (www.update-software.com/cochrane/).
Results: We identified 306 papers, of which 91 (30%) met the minimum inclusion criteria for a Cochrane EPOC review. The reasons for exclusion were uncontrolled before and after design (141/306; 46%) and inadequate interrupted time series (74/306; 24%) with fewer than three observations before and after the intervention. Most of the rejected interrupted time series (ITS) studies had only one or two data points before the intervention with many (up to 15) after it. Only 15 (40%) of the 38 included ITS studies had a statistical analysis and 11 of these used inappropriate statistical tests (e.g. t-test of pre- and post-intervention mean data) rather than analysis of time trends. Regression analysis of the proportion of included studies by year of publication did show a significant positive correlation (R2 = 0.7886). Nonetheless, of 47 papers published since 2000, only 19 (40%) met the minimum eligibility criteria.
Conclusions: The majority of evaluations used fundamentally flawed methodology. There is limited evidence of improvement over time. These problems could be resolved if researchers and referees of protocols or manuscripts implemented the EPOC methodology.
Keywords: prescribing interventions, controlled before and after studies, interrupted time series, segmented regression analysis, hospital antibiotic prescribing
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
In 1999, the British Society for Antimicrobial Chemotherapy and the Hospital Infection Society convened a Working Party on optimizing antibiotic prescribing in hospitals. The Working Party identified a substantial literature but no existing systematic review and has developed a protocol with the EPOC (Effective Practice and Organization of Care) Group of the Cochrane Collaboration. The primary aim was to systematically review the literature to identify interventions that alone, or in combination, are effective in improving antibiotic prescribing to hospital inpatients. In this paper, we document the reasons for exclusion of studies from the review and describe common methodological problems with statistical analysis in order to highlight key issues for investigators, editors and referees. We identified particular problems with interrupted time series (ITS) analysis. This is potentially the most practical of the rigorous methods for evaluation of a prescribing intervention in a hospital because it does not require measurement of practice at a control site.5 Nonetheless, it was clear from the review that authors, editors and referees were not aware of the minimum data requirements or of the correct method for statistical analysis of the intervention effect. In this paper we provide examples of common errors and show how to avoid them.
![]() |
Materials and methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Search strategy
MEDLINE was searched from 1980 onwards with the search terms: antibiotic use AND resistance, antibiotic guidelines, antibiotic guidelines AND implementation, antibiotic policies, antibiotic policies AND prescribing, antibiotic policies AND antibiotic use. All of the search terms were then re-used with antimicrobial instead of antibiotic; optimal antibiotic/antimicrobial prescribing/use. The electronic search was repeated in the Cochrane database of clinical trials, EMBASE and in the Cochrane EPOC register, compiled by searching MEDLINE (back to 1966), Health STAR (back to 1975), and EMBASE (back to 1980). The electronic searches were supplemented by a manual search of the bibliography from each identified paper. We included papers published before 1980 that were identified from the Cochrane EPOC register or by hand searching of reference lists from the papers identified by electronic searches. There were no language limitations.
Inclusion and exclusion criteria
Two authors reviewed all papers independently. We included randomized and controlled clinical trials, controlled before and after studies and interrupted time series that met the quality criteria of the Cochrane EPOC Group.6 We excluded uncontrolled before and after studies and interrupted time series studies with fewer than three data points before or after the intervention.
Statistical analysis of time series
The EPOC methods recommend segmented regression analysis to estimate the effect size for interrupted time series analyses, with reporting of change in level and slope of the regression line after the intervention (Figure 1). These methods were applied to any papers with sufficient numbers of data points, unless the original paper included an appropriate statistical analysis.
|
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
We identified 295 papers published from 1980 onwards and an additional 11 papers published from 1974 to 1979 that were identified by hand searching of reference lists from papers published after 1979. Of the 306 papers, 91 (30%) met the minimum inclusion criteria for a Cochrane EPOC review. The reasons for exclusion were uncontrolled before and after design (141/306; 46%) and inadequate interrupted time series (74/306; 24%) with fewer than three observations before and after the intervention (lists of excluded papers are available as Supplementary data at www.jac.oupjournals.org). There was some improvement in the proportion of eligible studies over time, but even so, the majority (28/47; 60%) of studies published from 2000 to 2003 were ineligible (Figure 2).
|
Analysis of interrupted time series studies with adequate pre- and post-intervention data
We identified 38 studies with a minimum of three data points before and after the intervention.744 Only four studies (10.5%) used the recommended segmented regression analysis9,17,40,43 to compare pre- and post-intervention data and two of these were from the same group.9,17 Eleven studies (29.0%) presented a statistical analysis of the differences between the means of data points before and after the intervention.10,12,14,15,18,21,23,25,36,37,41 The remaining 23 studies (60.5%) did not provide any statistical analysis of the interventions effect. The mean number of data points was 9.0 (range 340) before and 10.5 (range 329) after the intervention. The ratio of pre to post time points for each individual study was 0.96 (S.D. 0.62), confirming that the post phase was fractionally larger than the pre phase on average.
In an interrupted time series, comparison of the means of data before and after the intervention can either overestimate (Figure 3) or underestimate (Figure 4) the effect size. Overestimation occurs when there is a downward trend over time before the intervention begins (Figure 3). Conversely, underestimation occurs when there is an upward trend over time before the intervention begins (Figure 4). The other advantage of analysis of change in level and slope is that it provides information about the immediate impact of the intervention and about its sustained impact over time. The data in Figure 3 do show a significant reduction in vancomycin use immediately after the intervention but this was followed by a steady increase in use, so that vancomycin prescribing returned to pre-intervention levels within 11 months.
|
|
|
In the remaining five studies, analysis of means overestimated the effect size.10,23,25,36,37 The most striking examples occurred when the slope was decreasing significantly before the intervention and increasing after the intervention.10,23
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The commonest major problem was an uncontrolled before and after design. This is unacceptable because it does not address most of the important threats to validity of an experimental study. Without a control group, it is impossible to say whether or not any observed differences before and after the intervention would have happened anyway.46,47 A control group is also essential to establish that the intervention site is representative and hence to provide reassurance against any change simply being the result of regression to the mean.46
The interrupted time series is potentially a very powerful experimental design; in fact it is regarded as the strongest quasi-experimental design for evaluation of the longitudinal effects of prescribing interventions.5 Segmented regression analysis of time series data estimates how much an intervention changed an outcome, immediately and over time; instantly or with delay; and whether factors other than the intervention could explain the change. Two parameters define each segment of a time series: level and slope. A change in level is a jump or drop in the outcome after the intervention, which indicates an abrupt intervention effect. Change in level is measured from the difference between the last point pre-intervention and the first point post-intervention. A change in slope is defined by an increase or decrease in the slope after the intervention in comparison with the slope before the intervention and represents a gradual change over time. Depending on the direction of change in slope, it can either reverse (Figure 3) or enhance a change in level (Figure 4).
The Cochrane EPOC Data Collection Checklist recommends a minimum of three data points before and after the intervention in an interrupted time series.6 However, this is the absolute bare minimum required for a regression line. Adequate evaluation of seasonal variation requires a minimum of 24 monthly measures.5 Seasonal variation is likely to be an important issue in analyses of antibiotic prescribing but none of the ITS studies that we reviewed had 24 data points before and after the intervention, the closest was 29 before and 23 after.25 There also needs to be a sufficient number of observations and it is desirable to have a minimum of 100 observations at each time point to achieve an acceptable level of variability of the estimate at each time point.5
With adequate data in a time series, it is possible to eliminate key threats to validity, including seasonal changes in outcome and changes in the composition of the study population or in the measurement of outcome occurring at the time of the intervention. Separating intervention effects from other effects that occur at the same time requires a control. Ideally the control should be observation of the same outcome in a different group of subjects, without an intervention. However, control can also be achieved by including observations of a different outcome (e.g. prescribing of drugs other than antibiotics) in the same group of subjects.5
In conclusion, there are fundamental methodological flaws in the design and/or execution and/or the analysis of the results of the majority of published evaluations of interventions to improve antibiotic prescribing in hospitals. In some cases, this has caused investigators to reach unwarranted, or indeed erroneous, conclusions regarding the efficacies of the interventions. Researchers, referees and editors need to become familiar with the methodological standards that are readily accessible on the Cochrane EPOC Group website.6 Segmented regression analysis of interrupted time series is potentially a very powerful way to estimate the size of an effect at different time points, as well as changes in the trend of the effect over time.5 However, of 38 interrupted time series, only four (10.5%) used any form of regression analysis and none had enough data points to fully adjust for seasonal variation in antibiotic prescribing.
Supplementary data
Supplementary data for this paper are available at www.jac.oupjournals.org.
![]() |
Acknowledgements |
---|
Conflicts of interest
None declared (please note that conflicts of interest were considered for the Working Party as a whole).
![]() |
Footnotes |
---|
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
2 . House of Lords Select Committee on Science and Technology. (2001). Resistance to Antibiotics. Session 200001, 3rd Report, pp. 134. The Stationery Office, London, UK.
3 . European Union Conference. (1998). The Copenhagen Recommendations. Report from the Invitational EU Conference on The Microbial Threat, pp. 152. Ministry of Health, Ministry of Food, Agriculture and Fisheries, Copenhagen, Denmark http://www.im.dk/publikationer/micro98/index.htm.
4 . Standing Medical Advisory Committee, Subcommittee on Antibiotic Resistance. (1998). The Path of Least Resistance (Hart, M., Livermore, D. M. & Weinberg, J. R., Eds), pp. 1152. Department of Health, London, UK.
5 . Wagner, A. K., Soumerai, S. B., Zhang, F. et al. (2002). Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 27, 299309.[CrossRef][ISI][Medline]
6 . The Cochrane Effective Practice and Organisation of Care Group (EPOC). The Data Collection Checklist. (2002). Cochrane Effective Practice and Organisation of Care Group (EPOC) Methods Papers [Online.] http://www.epoc.uottawa.ca/ (22 April 2003, date last accessed).
7 . Abramowitz, P. W., Nold, E. G. & Hatfield, S. M. (1982). Use of clinical pharmacists to reduce cefamandole, cefoxitin and ticarcillin costs. American Journal of Hospital Pharmacy 39, 117680.[ISI][Medline]
8 . Adachi, W., Bolding, F. & Armstrong, R. (1997). Experience with vancomycin education and order sheet to limit vancomycin use. Hospital Pharmacy 32, 13703.
9 . Avorn, J., Soumerai, S. B., Taylor, W. et al. (1988). Reduction of incorrect antibiotic dosing through a structured educational order form. Archives of Internal Medicine 148, 17204.[Abstract]
10 . Belliveau, P. P., Rothman, A. L. & Maday, C. E. (1996). Limiting vancomycin use to combat vancomycin-resistant Enterococcus faecium. American Journal of Health Systems Pharmacy 53, 15705.
11
.
Bradley, S. J., Wilson, A. L. T., Allen, M. C. et al. (1999). The control of hyperendemic glycopeptide-resistant Enterococcus spp. on a haematology unit by changing antibiotic usage. Journal of Antimicrobial Chemotherapy 43, 2616.
12 . Bunz, D., Gupta, S. & Jewesson, P. (1990). Metronidazole cost containment: a two-stage intervention. Hospital Formulary 25, 116777.[ISI][Medline]
13 . Burke, C. E., Piper, J. & Holloway, W. (1997). Order form for restricting vancomycin prescribing. American Journal of Health Systems Pharmacy 54, 1893, 1897.
14
.
Climo, M. W., Israel, D. S. & Wong, E. S. (1998). Hospital-wide restriction of clindamycin: effect on the incidence of Clostridium difficile-associated diarrhea and cost. Annals of Internal Medicine 128, 98995.
15 . de Champs, C., Franchineau, P. & Gourgand, J. M. (1994). Clinical and bacteriological survey after change in aminoglycoside treatment to control an epidemic of Enterobacter cloacae. Journal of Hospital Infection 28, 21929.[ISI][Medline]
16 . Dempsey, C. L. (1995). Nursing home-acquired pneumonia: outcomes from a clinical process improvement program. Pharmacotherapy 15, 33S38S.[Medline]
17 . Everitt, D. E., Soumerai, S. B., Avorn, J. et al. (1990). Changing surgical antimicrobial prophylaxis practices through education targeted at senior department leaders. Infection Control and Hospital Epidemiology 11, 57883.[ISI][Medline]
18 . Gupta, S., Bachand, R. L. & Jewesson, P. J. (1989). Impact of a two-stage intervention program on cefazolin usage at a major teaching hospital. Hospital Formulary 24, 414, 46.
19 . Hampson, J. P., Corkhill, J. E., Murray, A. et al. (1988). Potential financial benefits of a local antibiotic policy. Pharmaceutical Journal Novermber 19, 6602.
20 . Hess, D. A., Mahoney, C. D., Johnson, P. N. et al. (1990). Integration of clinical and administrative strategies to reduce expenditures for antimicrobial agents. American Journal of Hospital Pharmacy 47, 58591.[ISI][Medline]
21 . Himmelberg, C. J., Pleasants, R. A., Weber, D. J. et al. (1991). Use of antimicrobial drugs in adults before and after removal of a restriction policy. American Journal of Public Health 48, 12207.
22 . Huber, S. L., Patry, R. A. & Hudson, H. D. (1982). Influencing drug use through prescribing restrictions. American Journal of Hospital Pharmacy 39, 1898901.[ISI][Medline]
23 . Inaraja, M. T., Paloma, J. M., Giraldez, J. et al. (1986). Computer-assisted antimicrobial-use monitoring. American Journal of Hospital Pharmacy 43, 66470.[ISI][Medline]
24 . Kolar, M. & Latal, T. (1999). Implementation of a practical antibiotic policy in the Czech Republic. Infection Control and Hospital Epidemiology 20, 4403.[ISI][Medline]
25 . Landman, D., Chockalingam, M. & Quale, J. M. (1999). Reduction in the incidence of methicillin-resistant Staphylococcus aureus and ceftazidime-resistant Klebsiella pneumoniae following changes in a hospital antibiotic formulary. Clinical Infectious Diseases 28, 10626.[ISI][Medline]
26 . Lautenbach, E., LaRosa, L. A., Marr, A. M. et al. (2003). Changes in the prevalence of vancomycin-resistant enterococci in response to antimicrobial formulary interventions: impact of progressive restrictions on use of vancomycin and third-generation cephalosporins. Clinical Infectious Diseases 36, 4406.[CrossRef][ISI][Medline]
27 . Lee, J., Carlson, J. A. & Chamberlain, M. A. (1995). A team approach to hospital formulary replacement. Diagnostic Microbiology and Infectious Disease 22, 23942.[CrossRef][ISI][Medline]
28 . McElnay, J. C., Scott, M. G., Sidara, J. Y. et al. (1995). Audit of antibiotic usage in medium-sized general hospital over an 11-year period. The impact of antibiotic policies. Pharmacy World and Science 17, 20713.[Medline]
29 . McGowan, J. E. & Finland, M. (1976). Effects of monitoring the usage of antibiotics: an interhospital comparison. Southern Medical Journal 69, 1935.[ISI][Medline]
30 . McGowan, J. E. & Finland, M. (1974). Usage of antibiotics in a general hospital: effect of requiring justification. Journal of Infectious Diseases 130, 1658.[ISI][Medline]
31 . McNulty, C., Logan, M., Donald, I. P. et al. (1997). Successful control of Clostridium difficile infection in an elderly care unit through use of a restrictive antibiotic policy. Journal of Antimicrobial Chemotherapy 40, 70711.[Abstract]
32 . Mercer, K. A., Chintalapudi, S. R. & Visconti, E. B. (1999). Impact of targeted antibiotic restriction on usage and cost in a community hospital. Journal of Pharmacy Technology 15, 7984.
33
.
Meyer, K. S., Urban, C., Eagan, J. A. et al. (1993). Nosocomial outbreak of Klebsiella infection resistant to late-generation cephalosporins. Annals of Internal Medicine 119, 3538.
34 . Patel, M. & Jackson, C. (1989). Targeted interventions on oral antibiotic expenditure. British Journal of Pharmaceutical Practice August, 3068.
35
.
Pear, S. M., Williamson, T. H., Bettin, K. M. et al. (1994). Decrease in nosocomial Clostridium difficile-associated diarrhea by restricting clindamycin use. Annals of Internal Medicine 120, 2727.
36 . Richardson, L. P., Wiseman, S. W., Melani, P. N. et al. (2000). Effectiveness of a vancomycin restriction policy in changing the prescribing patterns of house staff. Microbial Drug Resistance 6, 32730.[ISI][Medline]
37 . Saizy-Callaert, S., Causse, R., Furhman, C. et al. (2003). Impact of a multidisciplinary approach to the control of antibiotic prescription in a general hospital. Journal of Hospital Infection 53, 17782.[CrossRef][ISI][Medline]
38 . Salama, S., Rotstein, C. & Mandell, L. (1996). A multidisciplinary hospital-based antimicrobial use program: impact on hospital pharmacy expenditures and drug use. Canadian Journal of Infectious Diseases 7, 1049.
39 . Sirinavin, S., Suvanakoot, P., Sathapatayavongs, B. et al. (1998). Effect of antibiotic order form guiding rational use of expensive drugs on cost containment. Southeast Asian Journal of Tropical Medicine and Public Health 29, 63642.[Medline]
40 . Stevenson, R. C., Blackman, S. C., Williams, C. L. et al. (1988). Measuring the saving attributable to an antibiotic prescribing policy. Journal of Hospital Infection 11, 1625.[ISI][Medline]
41 . Toltzis, P., Yamashita, T., Vilt, L. et al. (1998). Antibiotic restriction does not alter endemic colonization with resistant gram-negative rods in a pediatric intensive care unit. Critical Care Medicine 26, 18939.[ISI][Medline]
42 . Wilson, J., Gordon, A., French, S. et al. (1991). The effectiveness of prescribers newsletters in influencing hospital drug expenditure. Hospital Pharmacy Practice May, 338.
43 . Woodward, R. S., Medoff, G., Smith, M. D. et al. (1987). Antibiotic cost savings from formulary restrictions and physician monitoring in a medical-school-affiliated hospital. American Journal of Medicine 83, 81723.[ISI][Medline]
44 . Young, E. J., Sewell, C. M., Koza, M. A. et al. (1985). Antibiotic resistance patterns during aminoglycoside restriction. American Journal of the Medical Sciences 290, 2237.[ISI][Medline]
45
.
Altman, D. G. (1994). The scandal of poor medical research: we need less research, better research, and research done for the right reasons. British Medical Journal 308, 2834.
46 . Cook, T. D. & Campbell, D. T. (1979). Quasi Experimentation. Design & Analysis Issues for Field Settings. Houghton Mifflin Company, Boston, MA, USA.
47 . Soumerai, S. B., McLaughlin, T. J. & Avorn, J. (1989). Improving drug prescribing in primary care: a critical analysis of the experimental literature. Milbank Quarterly 67, 268317.[ISI][Medline]