Evaluation of clinical guidelines for the management of end-stage renal disease in Europe: the EU BIOMED 1 Study

Editor's note

Craig R. Ramsay1,, Marion K. Campbell1, Diego Cantarovich3, Graeme Catto2, June Cody2, Conal Daly2, Catherine Delcroix3, Neil Edward2, Jeremy M. Grimshaw1, Henk W. van Hamersvelt5, Ian S. Henderson4, Izhar H. Khan2, Robert A. P. Koene5, Mitsokoulis Papadimitrou6, Eberhard Ritz7, Dimitri Tsakiris6 and Alison M. MacLeod2

1 Health Services Research Unit, Aberdeen, 2 Department of Medicine and Therapeutics and Renal Unit, Aberdeen, UK, 3 University of Nantes, Nantes, France, 4 Renal Unit, Ninewells Hospital, Dundee, UK, 5 Department of Nephrology, Academic Hospital, Nijmegen, The Netherlands, 6 Aristolelian University of Thessaloniki, Thessaloniki, Greece and 7 University of Heidelberg, Heidelberg, Germany



   Abstract
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Background. There are wide national and international variations in the management of patients with end-stage renal disease (ESRD). The aim of this study was to develop, harmonize, implement, and evaluate consensus-based clinical guidelines for the management of renal anaemia and renal bone disease in patients with ESRD, and for the prevention and management of cytomegalovirus disease in renal transplant recipients across six renal centres in Europe.

Methods. The trial was a prospective, multicentre, randomized balanced incomplete block design. Nephrologists from the six European renal units were randomized to develop and implement guidelines for two out of the three conditions and to act as a control for the third condition. Data were collected pre- (1 year) and post- (9 months) intervention on aspects of patient monitoring, management, and outcome.

Results. Eight hundred and twenty-nine dialysis patients from the six European dialysis centres were included in the study. Multivariate analysis (adjusting for case-mix and secular trends) showed a significant increase in the number of monitoring events in the guideline group compared with control group (6%, 95% CI, 1–11%). There was no concomitant increase in either appropriate management or the number of favourable patient outcomes.

Conclusions. In the first European collaboration on renal guidelines, the introduction of the guidelines improved the monitoring of the patients, but did not improve patient management or outcome. This study suggests the potential for creating clinical guidelines with the aim of standardizing treatment protocols across international boundaries, and improving the quality of the medical care provided.

Keywords: CAPD; clinical guidelines; cluster randomized trial; European guidelines; haemodialysis



   Introduction
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Currently there are wide national and international variations in the management of patients with end-stage renal disease (ESRD). The proportion of patients treated by haemodialysis or peritoneal dialysis varies between countries, as does the prescription of such specific treatments as erythropoietin and iron for correction of renal anaemia.

Although informal guidelines have been used by clinicians in their wards and clinics for many years, their effects on clinical practice and their ability to improve patient outcome have not generally been assessed. As more therapies become available, often at high cost, health services researchers are increasingly working with clinicians to develop and evaluate more formal clinical practice guidelines.

Such guidelines have been defined as ‘systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances’ [1] and are used by clinicians to improve the quality, effectiveness and cost-effectiveness of the medical care they provide by setting standards and reducing variations in clinical performance.

A recent systematic review of rigorous published evaluations of the introduction of clinical guidelines identified 91 studies [2]. Eighty-one of the 87 studies that measured the process of care observed improvements following the introduction of guidelines. Furthermore, 13 of the 17 papers that measured the outcome of care reported improvements. The review concluded that valid clinical guidelines, appropriately disseminated and implemented, can change practice and lead to improvements in patient outcome.

The systematic review did not identify any studies of guideline development and implementation in nephrology. Our group of nephrologists and health services researchers aimed to develop, implement, and evaluate consensus guidelines for the management of aspects of ESRD within the context of a European multicentre study funded by the European Union BIOMED programme.

We were aware of variations in the management of patients with ESRD and selected three for the development of specific guidelines. We selected conditions that were known to affect many patients and which had considerable resource implications. The conditions finally selected were the management of renal anaemia and renal bone disease, and the prevention and treatment of cytomegalovirus (CMV) infection in renal transplant recipients.



   Subjects and methods
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Study design
Nephrologists from six renal centres representing five European countries participated in the study. The six centres involved in the study were Aberdeen (UK), Dundee (UK), Heidelberg (Germany), Nantes (France), Nijmegen (the Netherlands), and Thessaloniki (Greece).

The study was a cluster randomized controlled trial using a balanced incomplete block design (Table 1Go) [3]. According to this design, each centre was randomized to develop and implement two out of three of the guidelines and to act as a control for the third. It was balanced in the sense that each centre acted as both a control and a guideline site and it is incomplete because a given centre did not manage both control and guideline patients for the same condition.


View this table:
[in this window]
[in a new window]
 
Table 1. Balanced incomplete block design

 

Intervention
Whenever possible, actions specified within the guidelines were based on published evidence and we agreed that where there was no consensus, this should be made explicit in the guideline. For more complex sections detailing indications for treatment and associated actions, algorithms were constructed for two of the guidelines (bone disease and CMV) in the form of decision trees. For sections on frequency of investigations and desirable outcomes, simple text was deemed sufficient.

At the end of the guideline development sessions, a meeting took place to highlight effective methods for the dissemination and implementation of the guideline in the local settings. There was no standard approach across centres and a variety of methods were used including regular local group meetings. Potential barriers to change and factors that influence the management of change were also discussed.

Data collection
A registration form was completed for all patients over the age of 16 years who were receiving dialysis during the initial 3-month period (‘stock’ patients) and subsequently for those patients who commenced dialysis during the study period. Data on biochemical indicators, demographic details, modality changes, and frequency of investigations were collected prospectively at 3-month intervals (called ‘updates‘) over the 2-year period of 1 September 1993 to 31 August 1995. Data were collected for 12 months before the guidelines were developed and implemented, and for 9 months afterwards. A 3-month period after the first year of data collection was used to develop and implement the three guidelines (Figure 1Go).



View larger version (13K):
[in this window]
[in a new window]
 
Fig. 1. Data collection.

 
Specific questions were devised (Table 2Go) that reflected the areas of care (monitoring, management or outcome) where we anticipated that the guideline could lead to improvements in clinical practice. For example, ferritin was not routinely monitored in all the centres at 3-monthly intervals and we anticipated that guidelines might improve this practice, whereas haematocrit was monitored regularly in all centres and improvement was not anticipated.


View this table:
[in this window]
[in a new window]
 
Table 2. The questions used to evaluate the guidelines

 
For each update period before and after guideline implementation, each area of care was scored according to whether it represented adherence to the guidelines or not. These scores were then summed to produce a total score for each patient before and after implementation. The balanced incomplete block design of the study meant that data from the same area of care (e.g. monitoring) could be combined across the three conditions to produce a single summary measure. For example, since ferritin was to be measured at least once every 3 months, if it was not measured during update period two, say, then it was scored zero out of one for that period. If ferritin was subsequently measured in all other update periods, then the pre-intervention score for that patient would have been three out of four.

Statistical analysis
Firstly, a univariate comparison of the unadjusted pre-guideline scores and post-guideline scores for the control vs guideline centres was undertaken. Secondly, a multivariate approach using generalized linear modelling was used to control for possible biases and clustering effects [4]. An analysis strategy was developed a priori that identified confounders requiring adjustment in the statistical model building. These confounders were mode of dialysis, time on renal replacement therapy, centre, phase of study (pre-/post-guideline), condition, and risk group for age and comorbidity [5], and included statistical interactions where appropriate. If the effect of such confounders was significant at the 5% level, they were included. The study had 80% power and 5% significance to detect an absolute difference of 5% in the monitoring of events, assuming the control sites had 60% appropriate events. This sample size calculation ignored the clustering effects, but this was offset by the design of the study.



   Results
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
There were 578 patients on dialysis at the start of the study and 261 new patients subsequently commenced dialysis during the remaining 21 months of the study—a total recruitment of 839 patients (Table 3Go). One hundred and eighty-six (22%) of the patients died during the study and 94 (11%) underwent renal transplantation.


View this table:
[in this window]
[in a new window]
 
Table 3. Details of the study population

 
Univariate analysis showed that the percentage of appropriately monitored events after the introduction of guidelines was higher in the guideline centres (79%) than in the controls (45%) (Table 4Go). However, due to a large imbalance in the pre-guideline values between the control and guideline centres, it is more appropriate to analyse changes in the two groups. The figures then showed only an 11% increase in the guideline centres and a 5% decrease in the control centres, thus providing some evidence that having guidelines increased the frequency of appropriate investigations. No change was observed in the ability to achieve the outcome targets between the control and guideline groups. Paradoxically, there was an improvement in the control centres for the management area of care.


View this table:
[in this window]
[in a new window]
 
Table 4. Unadjusted proportion of episodes of care complying with guideline

 
In the multivariate analysis, statistically significant improvement in the guideline centres for the patient monitoring area of care compared with control centres persisted (Table 5Go). This analysis adjusted for mode of dialysis, centre, guideline topic, and any trend in practice over time unrelated to the guidelines. The adjusted effect size was constant across the three conditions, showing a 6% increase in appropriate monitoring between the control and guideline groups (95% CI, 1–11%). There remained a baseline imbalance across the three guidelines between guideline and control centres.


View this table:
[in this window]
[in a new window]
 
Table 5. Results of the multivariate analysis

 
For the questions addressing the management of patients there was no evidence of a difference between control centres and those where guidelines were administered (-4%; 95% CI, -9 to +1%). This analysis adjusted for centre, guideline, risk group, and trend in practice over time. Similarly, there was no evidence of a difference in patient outcome (-7%; 95% CI, -14 to 1%) when adjustment was made for the centre, guideline, mode of dialysis, risk group, and any trend in practice over time.



   Discussion
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
We have shown that clinical guidelines for the management of ESRD can be created and disseminated within Europe. The process of creating the guidelines was very positive. It was important to have a group of well-motivated nephrologists and methodologists (statisticians and health services researchers) working together. The rotation of the clinical centres that were involved in the production of each guideline enabled all of the centres to actively participate, and this facilitated collaboration and the sense of ‘ownership’ of the project.

There was an increase in the number of monitoring events of 6% across the three guidelines, but no concomitant increase in the proportion of appropriately managed patients or improved patient outcomes. The appropriateness of the monitoring of the events was decided in advance by consensus. Thus the process of care rather than the outcome appeared to be affected.

The univariate analyses gave little indication of the effectiveness of the guidelines, but they were prone to biases. The multivariate analyses attempted to control for some of these biases by adjusting for factors, such as case mix and the numbers of patients per centre, before the guideline effect was tested. As anticipated there was a significant centre effect [6] and a significant difference between the three guidelines. Even adjusting for a number of confounders, there was still a baseline imbalance in the monitoring and outcome areas of patient care. Nevertheless, there appeared to be a statistically significant increase in the number of events that were monitored in the guideline centres, which was consistent across all three guidelines.

It is common to evaluate clinical interventions by analysing their effect on the three areas of care: structure (monitoring), management, and outcome. It is hypothesized that if you can firstly change process of care, then management of the patients will change and ultimately their outcomes will improve.

The design of the study (a balanced incomplete block) was an attempt to reduce the ‘Hawthorne effect’ in which an improvement in performance results from taking part in research. As with all cluster randomized trials there was a risk of baseline imbalance in performance between the guideline and control centres [8]. Such an effect was observed in our trial for both the monitoring and outcome areas of care.

The questions by which the guidelines were to be evaluated had to be in areas where guidelines could improve clinical management, and hence had to be chosen carefully. The compliance for the anaemia management question was 95% before the guidelines were introduced and so there was little expectation that a possible guideline effect could be detected. There was a suggestion that the control centres had a larger improvement in patient outcomes when compared with the guideline centres, but control centres initially had poorer outcomes and hence the improvement in these centres may have been in response to some trend over time (regression to the mean). There is therefore some evidence that we may have been observing a ‘ceiling effect’ occurring in the guideline centres because the outcomes were already relatively good and hence difficult to improve; an effect observed previously in monitoring dialysis patients' outcomes [9]. Also, it is possible that the length of follow-up of only 9 months was too short to show an effect. Alternatively, the guidelines could have been genuinely ineffective.

Since the development of these guidelines, there have been significant advances in the field of ‘evidence-based medicine’. There is increasing emphasis being placed on integrating clinical expertise with the best available evidence from systematic literature searches and randomized trials [10]. Consensus-based guidelines require to have their evidence base strengthened such that they can provide evidence of clinical effectiveness, and this should be taken into account when assessing guideline evaluations.

Although the guidelines were developed to change clinical practice, their effectiveness will depend on how well they were implemented and hence the guidelines were developed with the clinicians that will ultimately use them. This was a well-recognized strategy to increase compliance although there is increasing evidence that this method is not as effective as first anticipated [11]. We also distributed copies of the guidelines to all staff involved in patient care and local meetings were organized to discuss the guidelines. More rigorous methods of implementation such as patient-specific reminders may have proven more effective [12].

This was the first European guideline implementation project in ESRD. The results have not been entirely successful, given that we were not able to show improved patient outcomes. There was, however, a positive increase in the monitoring of patients after the guidelines were introduced. We encountered several difficulties. Evidence-based medicine was in its infancy as regards ESRD, the Cochrane Renal group had not been formed, and there were no systematic reviews on which to base our guidelines. There were all the difficulties of clustered randomized trials including the ‘ceiling’ and Hawthorne effects. The centres varied in their performance at the start of the study, making the comparison between control and test sites difficult. Implementation of the guidelines was difficult and should be a major component of any future study. We have shown, however, that there was a willingness of clinicians and health services researchers to work across international boundaries and we agreed guidelines and made some changes in the delivery of patient care.



   Acknowledgments
 
This study was funded by the European Union BIOMED 1 programme. The Health Services Research Unit is funded by the Chief Scientist Office of the Scottish Executive Health Department. Opinions expressed are those of the authors and not necessarily those of the funding body.



   Notes
 
Please see also Editorial Comment by F. Locatelli et al., pp. 1284–1287.

Correspondence and offprint requests to: Craig R. Ramsay, Health Services Research Unit, University of Aberdeen, Foresterhill, Aberdeen AB25 2ZD, UK. Back



   References
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 

  1. Field MJ, Lohr KN. Guidelines for Clinical Practice: From Development to Use. Institute of Medicine, Washington, DC, 1992
  2. Effective Health Care Bulletin. Implementing Clinical Guidelines. Can Guidelines be Used to Improve Clinical Practice? Bulletin No 8, University of Leeds, Leeds, 1994
  3. Cook TD, Campbell DT. Quasi-experimentation. Design and Analysis Issues for Field Settings. Rand McNally College Publishing Company, Chicago, 1979
  4. McCullagh P, Nelder JA. Generalized Linear Models Chapman and Hall, London, 1989
  5. Khan IH, Campbell MK, Cantarovich D et al. Comparing outcomes in renal replacement therapy: how should we correct for case mix? Am J Kidney Dis1998 31: 473–478[ISI][Medline]
  6. Khan IH, Campbell MK, Cantarovich D et al. Survival on renal replacement therapy in Europe: is there a ‘centre effect’? Nephrol Dial Transplant1996; 11: 300–307[Abstract]
  7. Donabedian A. Evaluating the quality of medical care. Millbank Memorial Fund Quarterly1966 44:166–206
  8. Donner A. Some aspects of the design and analysis of cluster randomisation trials. Appl Statist1998; 47: 95–113
  9. Stevens G, Meier BJ, McCarthy JT, Carlson D. Using comparative facility review to improve dialysis patient outcomes. Dial Transplant<1996; 25: 568–578[ISI]
  10. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence Based Medicine: How to Practice and Teach EBM. Churchill Livingstone, New York, 1997
  11. Grimshaw J, Freemantle N, Wallace S et al. Developing and implementing clinical practice guidelines. Qual Health Care1995; 4: 55–64[Medline]
  12. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance: a systematic review of the effect of continuing medical education strategies. J Am Med Assoc1995; 274: 700–705[Abstract]
Received for publication: 13. 4.99
Accepted in revised form: 9. 2.00