Room for improvement: a systematic review of the quality of evaluations of interventions to improve hospital antibiotic prescribing

Craig Ramsay1, Erwin Brown2, Giles Hartman3 and Peter Davey4,* on behalf of the joint BSAC/HIS Working Party on Optimization of Antibiotic Prescribing in Hospitals

1 Health Services Research Unit, University of Aberdeen, Aberdeen; 2 Department of Microbiology, Frenchay Hospital, Bristol; 3 Department of Pathology and Microbiology, University of Bristol, Bristol; 4 MEMO, Department of Clinical Pharmacology, University of Dundee, Dundee DD1 9SY, UK

Received 18 June 2003; returned 17 August 2003; revised 28 August 2003; accepted 28 August 2003


    Abstract
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
Introduction: In 1999, the British Society for Antimicrobial Chemotherapy (BSAC) and Hospital Infection Society (HIS) convened a working party on optimization of antibiotic prescribing in hospitals. This study was undertaken in order to evaluate the current evidence base on the effectiveness of interventions to change antibiotic prescribing to hospital inpatients.

Methods: We have systematically reviewed the literature from 1980 to identify interventions that alone, or in combination, are effective in improving antibiotic prescribing to hospital inpatients. The protocol was peer reviewed and has been published by the Effective Practice and Organization of Care (EPOC) Group of the Cochrane Collaboration (www.update-software.com/cochrane/).

Results: We identified 306 papers, of which 91 (30%) met the minimum inclusion criteria for a Cochrane EPOC review. The reasons for exclusion were uncontrolled before and after design (141/306; 46%) and inadequate interrupted time series (74/306; 24%) with fewer than three observations before and after the intervention. Most of the rejected interrupted time series (ITS) studies had only one or two data points before the intervention with many (up to 15) after it. Only 15 (40%) of the 38 included ITS studies had a statistical analysis and 11 of these used inappropriate statistical tests (e.g. t-test of pre- and post-intervention mean data) rather than analysis of time trends. Regression analysis of the proportion of included studies by year of publication did show a significant positive correlation (R2 = 0.7886). Nonetheless, of 47 papers published since 2000, only 19 (40%) met the minimum eligibility criteria.

Conclusions: The majority of evaluations used fundamentally flawed methodology. There is limited evidence of improvement over time. These problems could be resolved if researchers and referees of protocols or manuscripts implemented the EPOC methodology.

Keywords: prescribing interventions, controlled before and after studies, interrupted time series, segmented regression analysis, hospital antibiotic prescribing


    Introduction
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
Despite strenuous efforts to control antibiotic usage and to promote optimal prescribing, practitioners continue to prescribe inappropriately; it is estimated that up to 50% of antibiotic usage in hospitals is inappropriate.1 Recently, a number of reports have proposed a range of measures designed to address the problem of increasing resistance.14 Common to all the recommendations is the challenge to reduce inappropriate antibiotic prescribing, the implication being that antibiotic resistance is largely a consequence of the selective pressures of antibiotic usage and that reducing these pressures by the judicious administration of antibiotics will facilitate a return of susceptible bacteria or, at least, will prevent or slow the pace of the emergence of resistant strains.

In 1999, the British Society for Antimicrobial Chemotherapy and the Hospital Infection Society convened a Working Party on optimizing antibiotic prescribing in hospitals. The Working Party identified a substantial literature but no existing systematic review and has developed a protocol with the EPOC (Effective Practice and Organization of Care) Group of the Cochrane Collaboration. The primary aim was to systematically review the literature to identify interventions that alone, or in combination, are effective in improving antibiotic prescribing to hospital inpatients. In this paper, we document the reasons for exclusion of studies from the review and describe common methodological problems with statistical analysis in order to highlight key issues for investigators, editors and referees. We identified particular problems with interrupted time series (ITS) analysis. This is potentially the most practical of the rigorous methods for evaluation of a prescribing intervention in a hospital because it does not require measurement of practice at a control site.5 Nonetheless, it was clear from the review that authors, editors and referees were not aware of the minimum data requirements or of the correct method for statistical analysis of the intervention effect. In this paper we provide examples of common errors and show how to avoid them.


    Materials and methods
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
The full protocol for the systematic review is available on the Cochrane website (www.update-software.com/cochrane/).

Search strategy

MEDLINE was searched from 1980 onwards with the search terms: ‘antibiotic use AND resistance’, ‘antibiotic guidelines’, ‘antibiotic guidelines AND implementation’, ‘antibiotic policies’, ‘antibiotic policies AND prescribing’, ‘antibiotic policies AND antibiotic use’. All of the search terms were then re-used with ‘antimicrobial’ instead of ‘antibiotic’; ‘optimal antibiotic/antimicrobial prescribing/use’. The electronic search was repeated in the Cochrane database of clinical trials, EMBASE and in the Cochrane EPOC register, compiled by searching MEDLINE (back to 1966), Health STAR (back to 1975), and EMBASE (back to 1980). The electronic searches were supplemented by a manual search of the bibliography from each identified paper. We included papers published before 1980 that were identified from the Cochrane EPOC register or by hand searching of reference lists from the papers identified by electronic searches. There were no language limitations.

Inclusion and exclusion criteria

Two authors reviewed all papers independently. We included randomized and controlled clinical trials, controlled before and after studies and interrupted time series that met the quality criteria of the Cochrane EPOC Group.6 We excluded uncontrolled before and after studies and interrupted time series studies with fewer than three data points before or after the intervention.

Statistical analysis of time series

The EPOC methods recommend segmented regression analysis to estimate the effect size for interrupted time series analyses, with reporting of change in level and slope of the regression line after the intervention (Figure 1). These methods were applied to any papers with sufficient numbers of data points, unless the original paper included an appropriate statistical analysis.



View larger version (6K):
[in this window]
[in a new window]
 
Figure 1. Graphic illustration of segmented regression analysis of interrupted time series data.

 

    Results
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
Reasons for exclusion of papers from the review

We identified 295 papers published from 1980 onwards and an additional 11 papers published from 1974 to 1979 that were identified by hand searching of reference lists from papers published after 1979. Of the 306 papers, 91 (30%) met the minimum inclusion criteria for a Cochrane EPOC review. The reasons for exclusion were uncontrolled before and after design (141/306; 46%) and inadequate interrupted time series (74/306; 24%) with fewer than three observations before and after the intervention (lists of excluded papers are available as Supplementary data at www.jac.oupjournals.org). There was some improvement in the proportion of eligible studies over time, but even so, the majority (28/47; 60%) of studies published from 2000 to 2003 were ineligible (Figure 2).



View larger version (7K):
[in this window]
[in a new window]
 
Figure 2. Proportion of papers that met the minimum eligibility criteria for the Cochrane review of interventions to improve hospital antibiotic prescribing, by year of publication.

 
Inadequate number of pre-intervention time points was the commonest reason for rejection of interrupted time series studies. The number of pre- and post-intervention points was unclear for three studies. Of the remaining 71 studies, only nine (13%) had three or more pre-intervention points. In contrast, 39 (55%) of the 71 studies had three or more post-intervention points, the maximum being 24. The ratio of pre- to post-intervention time points for each individual study was 0.67 (S.D. 0.94), confirming that the post phase was usually larger than the pre phase on average.

Analysis of interrupted time series studies with adequate pre- and post-intervention data

We identified 38 studies with a minimum of three data points before and after the intervention.744 Only four studies (10.5%) used the recommended segmented regression analysis9,17,40,43 to compare pre- and post-intervention data and two of these were from the same group.9,17 Eleven studies (29.0%) presented a statistical analysis of the differences between the means of data points before and after the intervention.10,12,14,15,18,21,23,25,36,37,41 The remaining 23 studies (60.5%) did not provide any statistical analysis of the intervention’s effect. The mean number of data points was 9.0 (range 3–40) before and 10.5 (range 3–29) after the intervention. The ratio of pre to post time points for each individual study was 0.96 (S.D. 0.62), confirming that the post phase was fractionally larger than the pre phase on average.

In an interrupted time series, comparison of the means of data before and after the intervention can either overestimate (Figure 3) or underestimate (Figure 4) the effect size. Overestimation occurs when there is a downward trend over time before the intervention begins (Figure 3). Conversely, underestimation occurs when there is an upward trend over time before the intervention begins (Figure 4). The other advantage of analysis of change in level and slope is that it provides information about the immediate impact of the intervention and about its sustained impact over time. The data in Figure 3 do show a significant reduction in vancomycin use immediately after the intervention but this was followed by a steady increase in use, so that vancomycin prescribing returned to pre-intervention levels within 11 months.



View larger version (14K):
[in this window]
[in a new window]
 
Figure 3. An example of an interrupted time series in which the effect of the interventions is overestimated by analysis of mean data before and after the intervention.10

 


View larger version (17K):
[in this window]
[in a new window]
 
Figure 4. An example of an interrupted time series in which the effect of the interventions is underestimated by analysis of mean data before and after the intervention.33

 
The results of statistical analysis of changes in mean, level and slope have been compared for the 11 studies that presented a statistical analysis of means (Table 1). In three studies, there was a significant decrease in the mean, level and slope after the intervention.14,15,41 However, analysis of means substantially underestimated the effect size in all three studies. There was an immediate change in level that exceeded the changes in mean and was then followed by a significant, sustained change in slope. The main reason that analysis of means underestimated the effect size was that the slope was increasing significantly before the intervention in all three studies (Table 1).


View this table:
[in this window]
[in a new window]
 
Table 1. Comparison of changes in mean, level and slope for interrupted time series studies that presented a statistical analysis of change in mean
 
There was good agreement between analysis of means and segmented regression analysis in three studies. In one, the intervention had no significant effect.21 In the other two, there was no significant change in slope before the intervention and it was followed by a significant change in level, which was similar in size to the change in means.12,18

In the remaining five studies, analysis of means overestimated the effect size.10,23,25,36,37 The most striking examples occurred when the slope was decreasing significantly before the intervention and increasing after the intervention.10,23


    Discussion
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
Nearly 10 years ago, Douglas Altman described the "scandal of poor medical research", stimulated by the high proportion of published papers with fundamentally flawed designs and inadequate statistical analysis.45 It is ironic that Altman emphasized the gravity of the situation with a metaphor based on prescribing: "What would we think about a doctor who uses the wrong treatment, or the right treatment wrongly?" The literature that we have reviewed starts from the premise that doctors’ prescribing needs to be improved, yet we have shown that even in the last 3 years, nearly two-thirds of the published evaluations of hospital antibiotic prescribing were fundamentally flawed. We should emphasize that the 91 studies that are eligible for the review only meet minimum criteria and it is likely that several will be excluded from the final review because of major methodological flaws.

The commonest major problem was an uncontrolled before and after design. This is unacceptable because it does not address most of the important threats to validity of an experimental study. Without a control group, it is impossible to say whether or not any observed differences before and after the intervention would have happened anyway.46,47 A control group is also essential to establish that the intervention site is representative and hence to provide reassurance against any change simply being the result of regression to the mean.46

The interrupted time series is potentially a very powerful experimental design; in fact it is regarded as the strongest quasi-experimental design for evaluation of the longitudinal effects of prescribing interventions.5 Segmented regression analysis of time series data estimates how much an intervention changed an outcome, immediately and over time; instantly or with delay; and whether factors other than the intervention could explain the change. Two parameters define each segment of a time series: level and slope. A change in level is a jump or drop in the outcome after the intervention, which indicates an abrupt intervention effect. Change in level is measured from the difference between the last point pre-intervention and the first point post-intervention. A change in slope is defined by an increase or decrease in the slope after the intervention in comparison with the slope before the intervention and represents a gradual change over time. Depending on the direction of change in slope, it can either reverse (Figure 3) or enhance a change in level (Figure 4).

The Cochrane EPOC Data Collection Checklist recommends a minimum of three data points before and after the intervention in an interrupted time series.6 However, this is the absolute bare minimum required for a regression line. Adequate evaluation of seasonal variation requires a minimum of 24 monthly measures.5 Seasonal variation is likely to be an important issue in analyses of antibiotic prescribing but none of the ITS studies that we reviewed had 24 data points before and after the intervention, the closest was 29 before and 23 after.25 There also needs to be a sufficient number of observations and it is desirable to have a minimum of 100 observations at each time point to achieve an acceptable level of variability of the estimate at each time point.5

With adequate data in a time series, it is possible to eliminate key threats to validity, including seasonal changes in outcome and changes in the composition of the study population or in the measurement of outcome occurring at the time of the intervention. Separating intervention effects from other effects that occur at the same time requires a control. Ideally the control should be observation of the same outcome in a different group of subjects, without an intervention. However, control can also be achieved by including observations of a different outcome (e.g. prescribing of drugs other than antibiotics) in the same group of subjects.5

In conclusion, there are fundamental methodological flaws in the design and/or execution and/or the analysis of the results of the majority of published evaluations of interventions to improve antibiotic prescribing in hospitals. In some cases, this has caused investigators to reach unwarranted, or indeed erroneous, conclusions regarding the efficacies of the interventions. Researchers, referees and editors need to become familiar with the methodological standards that are readily accessible on the Cochrane EPOC Group website.6 Segmented regression analysis of interrupted time series is potentially a very powerful way to estimate the size of an effect at different time points, as well as changes in the trend of the effect over time.5 However, of 38 interrupted time series, only four (10.5%) used any form of regression analysis and none had enough data points to fully adjust for seasonal variation in antibiotic prescribing.

Supplementary data

Supplementary data for this paper are available at www.jac.oupjournals.org.


    Acknowledgements
 
The Health Services Research Unit is funded by the Chief Scientist Office of the Scottish Executive Health Department. However, the views expressed are those of the authors and not necessarily those of the funding body. The Medicines Monitoring Unit and the Health Services Research Unit are both part of the MRC Health Services Research Collaboration, lead Centre the Department of Social Medicine, University of Bristol.

Conflicts of interest

None declared (please note that conflicts of interest were considered for the Working Party as a whole).


    Footnotes
 
* Corresponding author. Tel: +44-1382-632575; Fax: +44-1382-642637; E-mail: peter{at}memo.dundee.ac.uk Back


    References
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
1 . House of Lords Select Committee on Science and Technology. (1998). Resistance to Antibiotics and Other Antimicrobial Agents. Session 1997–98, 7th Report, pp. 1–108. The Stationery Office, London, UK.

2 . House of Lords Select Committee on Science and Technology. (2001). Resistance to Antibiotics. Session 2000–01, 3rd Report, pp. 1–34. The Stationery Office, London, UK.

3 . European Union Conference. (1998). The Copenhagen Recommendations. Report from the Invitational EU Conference on The Microbial Threat, pp. 1–52. Ministry of Health, Ministry of Food, Agriculture and Fisheries, Copenhagen, Denmark http://www.im.dk/publikationer/micro98/index.htm.

4 . Standing Medical Advisory Committee, Subcommittee on Antibiotic Resistance. (1998). The Path of Least Resistance (Hart, M., Livermore, D. M. & Weinberg, J. R., Eds), pp. 1–152. Department of Health, London, UK.

5 . Wagner, A. K., Soumerai, S. B., Zhang, F. et al. (2002). Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 27, 299–309.[CrossRef][ISI][Medline]

6 . The Cochrane Effective Practice and Organisation of Care Group (EPOC). The Data Collection Checklist. (2002). Cochrane Effective Practice and Organisation of Care Group (EPOC) Methods Papers [Online.] http://www.epoc.uottawa.ca/ (22 April 2003, date last accessed).

7 . Abramowitz, P. W., Nold, E. G. & Hatfield, S. M. (1982). Use of clinical pharmacists to reduce cefamandole, cefoxitin and ticarcillin costs. American Journal of Hospital Pharmacy 39, 1176–80.[ISI][Medline]

8 . Adachi, W., Bolding, F. & Armstrong, R. (1997). Experience with vancomycin education and order sheet to limit vancomycin use. Hospital Pharmacy 32, 1370–3.

9 . Avorn, J., Soumerai, S. B., Taylor, W. et al. (1988). Reduction of incorrect antibiotic dosing through a structured educational order form. Archives of Internal Medicine 148, 1720–4.[Abstract]

10 . Belliveau, P. P., Rothman, A. L. & Maday, C. E. (1996). Limiting vancomycin use to combat vancomycin-resistant Enterococcus faecium. American Journal of Health Systems Pharmacy 53, 1570–5.

11 . Bradley, S. J., Wilson, A. L. T., Allen, M. C. et al. (1999). The control of hyperendemic glycopeptide-resistant Enterococcus spp. on a haematology unit by changing antibiotic usage. Journal of Antimicrobial Chemotherapy 43, 261–6.[Abstract/Free Full Text]

12 . Bunz, D., Gupta, S. & Jewesson, P. (1990). Metronidazole cost containment: a two-stage intervention. Hospital Formulary 25, 1167–77.[ISI][Medline]

13 . Burke, C. E., Piper, J. & Holloway, W. (1997). Order form for restricting vancomycin prescribing. American Journal of Health Systems Pharmacy 54, 1893, 1897.

14 . Climo, M. W., Israel, D. S. & Wong, E. S. (1998). Hospital-wide restriction of clindamycin: effect on the incidence of Clostridium difficile-associated diarrhea and cost. Annals of Internal Medicine 128, 989–95.[Abstract/Free Full Text]

15 . de Champs, C., Franchineau, P. & Gourgand, J. M. (1994). Clinical and bacteriological survey after change in aminoglycoside treatment to control an epidemic of Enterobacter cloacae. Journal of Hospital Infection 28, 219–29.[ISI][Medline]

16 . Dempsey, C. L. (1995). Nursing home-acquired pneumonia: outcomes from a clinical process improvement program. Pharmacotherapy 15, 33S–38S.[Medline]

17 . Everitt, D. E., Soumerai, S. B., Avorn, J. et al. (1990). Changing surgical antimicrobial prophylaxis practices through education targeted at senior department leaders. Infection Control and Hospital Epidemiology 11, 578–83.[ISI][Medline]

18 . Gupta, S., Bachand, R. L. & Jewesson, P. J. (1989). Impact of a two-stage intervention program on cefazolin usage at a major teaching hospital. Hospital Formulary 24, 41–4, 46.

19 . Hampson, J. P., Corkhill, J. E., Murray, A. et al. (1988). Potential financial benefits of a local antibiotic policy. Pharmaceutical Journal Novermber 19, 660–2.

20 . Hess, D. A., Mahoney, C. D., Johnson, P. N. et al. (1990). Integration of clinical and administrative strategies to reduce expenditures for antimicrobial agents. American Journal of Hospital Pharmacy 47, 585–91.[ISI][Medline]

21 . Himmelberg, C. J., Pleasants, R. A., Weber, D. J. et al. (1991). Use of antimicrobial drugs in adults before and after removal of a restriction policy. American Journal of Public Health 48, 1220–7.

22 . Huber, S. L., Patry, R. A. & Hudson, H. D. (1982). Influencing drug use through prescribing restrictions. American Journal of Hospital Pharmacy 39, 1898–901.[ISI][Medline]

23 . Inaraja, M. T., Paloma, J. M., Giraldez, J. et al. (1986). Computer-assisted antimicrobial-use monitoring. American Journal of Hospital Pharmacy 43, 664–70.[ISI][Medline]

24 . Kolar, M. & Latal, T. (1999). Implementation of a practical antibiotic policy in the Czech Republic. Infection Control and Hospital Epidemiology 20, 440–3.[ISI][Medline]

25 . Landman, D., Chockalingam, M. & Quale, J. M. (1999). Reduction in the incidence of methicillin-resistant Staphylococcus aureus and ceftazidime-resistant Klebsiella pneumoniae following changes in a hospital antibiotic formulary. Clinical Infectious Diseases 28, 1062–6.[ISI][Medline]

26 . Lautenbach, E., LaRosa, L. A., Marr, A. M. et al. (2003). Changes in the prevalence of vancomycin-resistant enterococci in response to antimicrobial formulary interventions: impact of progressive restrictions on use of vancomycin and third-generation cephalosporins. Clinical Infectious Diseases 36, 440–6.[CrossRef][ISI][Medline]

27 . Lee, J., Carlson, J. A. & Chamberlain, M. A. (1995). A team approach to hospital formulary replacement. Diagnostic Microbiology and Infectious Disease 22, 239–42.[CrossRef][ISI][Medline]

28 . McElnay, J. C., Scott, M. G., Sidara, J. Y. et al. (1995). Audit of antibiotic usage in medium-sized general hospital over an 11-year period. The impact of antibiotic policies. Pharmacy World and Science 17, 207–13.[Medline]

29 . McGowan, J. E. & Finland, M. (1976). Effects of monitoring the usage of antibiotics: an interhospital comparison. Southern Medical Journal 69, 193–5.[ISI][Medline]

30 . McGowan, J. E. & Finland, M. (1974). Usage of antibiotics in a general hospital: effect of requiring justification. Journal of Infectious Diseases 130, 165–8.[ISI][Medline]

31 . McNulty, C., Logan, M., Donald, I. P. et al. (1997). Successful control of Clostridium difficile infection in an elderly care unit through use of a restrictive antibiotic policy. Journal of Antimicrobial Chemotherapy 40, 707–11.[Abstract]

32 . Mercer, K. A., Chintalapudi, S. R. & Visconti, E. B. (1999). Impact of targeted antibiotic restriction on usage and cost in a community hospital. Journal of Pharmacy Technology 15, 79–84.

33 . Meyer, K. S., Urban, C., Eagan, J. A. et al. (1993). Nosocomial outbreak of Klebsiella infection resistant to late-generation cephalosporins. Annals of Internal Medicine 119, 353–8.[Abstract/Free Full Text]

34 . Patel, M. & Jackson, C. (1989). Targeted interventions on oral antibiotic expenditure. British Journal of Pharmaceutical Practice August, 306–8.

35 . Pear, S. M., Williamson, T. H., Bettin, K. M. et al. (1994). Decrease in nosocomial Clostridium difficile-associated diarrhea by restricting clindamycin use. Annals of Internal Medicine 120, 272–7.[Abstract/Free Full Text]

36 . Richardson, L. P., Wiseman, S. W., Melani, P. N. et al. (2000). Effectiveness of a vancomycin restriction policy in changing the prescribing patterns of house staff. Microbial Drug Resistance 6, 327–30.[ISI][Medline]

37 . Saizy-Callaert, S., Causse, R., Furhman, C. et al. (2003). Impact of a multidisciplinary approach to the control of antibiotic prescription in a general hospital. Journal of Hospital Infection 53, 177–82.[CrossRef][ISI][Medline]

38 . Salama, S., Rotstein, C. & Mandell, L. (1996). A multidisciplinary hospital-based antimicrobial use program: impact on hospital pharmacy expenditures and drug use. Canadian Journal of Infectious Diseases 7, 104–9.

39 . Sirinavin, S., Suvanakoot, P., Sathapatayavongs, B. et al. (1998). Effect of antibiotic order form guiding rational use of expensive drugs on cost containment. Southeast Asian Journal of Tropical Medicine and Public Health 29, 636–42.[Medline]

40 . Stevenson, R. C., Blackman, S. C., Williams, C. L. et al. (1988). Measuring the saving attributable to an antibiotic prescribing policy. Journal of Hospital Infection 11, 16–25.[ISI][Medline]

41 . Toltzis, P., Yamashita, T., Vilt, L. et al. (1998). Antibiotic restriction does not alter endemic colonization with resistant gram-negative rods in a pediatric intensive care unit. Critical Care Medicine 26, 1893–9.[ISI][Medline]

42 . Wilson, J., Gordon, A., French, S. et al. (1991). The effectiveness of prescribers newsletters in influencing hospital drug expenditure. Hospital Pharmacy Practice May, 33–8.

43 . Woodward, R. S., Medoff, G., Smith, M. D. et al. (1987). Antibiotic cost savings from formulary restrictions and physician monitoring in a medical-school-affiliated hospital. American Journal of Medicine 83, 817–23.[ISI][Medline]

44 . Young, E. J., Sewell, C. M., Koza, M. A. et al. (1985). Antibiotic resistance patterns during aminoglycoside restriction. American Journal of the Medical Sciences 290, 223–7.[ISI][Medline]

45 . Altman, D. G. (1994). The scandal of poor medical research: we need less research, better research, and research done for the right reasons. British Medical Journal 308, 283–4.[Free Full Text]

46 . Cook, T. D. & Campbell, D. T. (1979). Quasi Experimentation. Design & Analysis Issues for Field Settings. Houghton Mifflin Company, Boston, MA, USA.

47 . Soumerai, S. B., McLaughlin, T. J. & Avorn, J. (1989). Improving drug prescribing in primary care: a critical analysis of the experimental literature. Milbank Quarterly 67, 268–317.[ISI][Medline]