Evidence-based medicine in nephrology: identifying and critically appraising the literature

Marion K. Campbell,1, Conal Daly2, Sheila A. Wallace1, D. June Cody2, Cam Donaldson3,5, Adrian M. Grant1, Izhar H. Khan2, Paul Lawrence4, Luke Vale1,3 and Alison M. MacLeod2

1 Health Services Research Unit, 2 Department of Medicine and Therapeutics, 3 Health Economics Research Unit, 4 Medical School Library, University of Aberdeen, Aberdeen, UK, 5 Departments of Economics and Community Health Sciences, University of Calgary, Canada



   Abstract
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Background. Uncertainties about best management of end-stage renal disease (ESRD) are reflected in wide variations in practice. Systematic reviews aim to reduce uncertainty by strengthening the evidence base for clinical practice, allowing estimation of the benefits and risks of particular interventions, whilst minimizing the potential for bias. This paper describes the methods and conduct of six systematic reviews of aspects of the management of ESRD, and the yield in terms of trials found.

Methods. Our methodology was based on that recommended by the Cochrane Collaboration (an international initiative set up to perform and disseminate systematic reviews of health care). It involved a systematic search of electronic databases and bibliographic reference lists, together with handsearching of Kidney International for studies relevant to the management of ESRD, followed by a systematic assessment of study quality.

Results. Around 12 000 abstracts were assessed which had been identified from electronic sources. Of these, 2085 (18%) were deemed to be reports of possible randomized or quasi-randomized controlled trials relevant to the management of ESRD. Three hundred and forty were relevant to the six specific reviews, and after assessment of the full manuscripts, 39 studies were finally included in our reviews. Reports of a further nine trials, which were identified from other sources, were also included. The broad search adopted allowed the parallel development of a register of trials of all aspects of the management of ESRD.

Conclusions. This study has demonstrated that the methodology of systematic reviews, as promoted by the Cochrane Renal Group, is feasible but has significant resource implications. The development of a register of randomized controlled trials (RCTs) related to the management of ESRD will facilitate this form of research in the future.

Keywords: end-stage renal disease; randomized controlled trials; systematic reviews



   Introduction
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
There are marked variations in the management of patients with end-stage renal disease (ESRD). For example, the proportion of patients treated by haemodialysis or peritoneal dialysis varies widely between centres in Europe, as does the use of specific components of treatment such as type of haemodialysis membrane (synthetic or cellulose), and type of peritoneal dialysis delivery/transfer set (Y-set or non-Y-set). Such variations often reflect uncertainty about the true benefits and risks of one treatment choice over another. Increasingly, both providers and consumers of health care are demanding information on the most effective and efficient therapies which can then be used to develop standards and guidelines for high-quality clinical practice.

The randomized controlled trial (RCT) is now widely accepted as the ‘gold standard’ method for assessing the relative effectiveness of alternative treatments [1,2]. This approach to evaluation ensures that selection and other common sources of bias are minimized. The interpretation of the results of individual trials may, however, be difficult, as few are of sufficient size to rule out chance as an explanation for observed differences between treatments. In addition, entry criteria for inclusion of patients into individual trials may be too restrictive to allow confident extrapolation of the results to a particular patient. Considering all relevant trials together should redress the limitations of individual trials.

Literature reviews can, however, potentially introduce other sources of bias such as through failure to identify all trials or to take into account methodological quality. Within what has become known as a ‘systematic review’, all relevant trials are identified in a systematic and reproducible manner, the methodological quality of individual trials is assessed objectively, and data are abstracted and synthesized according to a common protocol.

Whilst a systematic review of all relevant trials will address which treatment option is more effective, it is also important to assess the impact on resource use and hence efficiency. If, for example, one method of delivering care to patients with ESRD is shown to be more effective and less costly than its alternative, it should then be recommended as the treatment of choice. If, however, the more effective option is more costly, the magnitude of this extra cost may help to decide whether it is justified in terms of the improvement in patient outcome.

This paper describes the methodology used to conduct six systematic reviews of both the effectiveness and efficiency of aspects of the management of ESRD (excluding transplantation) commissioned by the UK NHS Research and Development Health Technology Assessment Programme. These are some of the first systematic reviews to be undertaken in ESRD and will contribute to the work of the Cochrane Renal Group [3].



   Methods
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Selection of topics
Because of the breadth of the subject area we concentrated on six key decision points where there is both uncertainty as to best clinical practice and important resource implications (Table 1Go).


View this table:
[in this window]
[in a new window]
 
Table 1. Choice of topics

 

Protocols
A protocol describing the proposed process was formulated for each topic which explicitly described the objectives of the review, the clinical and methodological criteria required of studies for inclusion, the outcome measures of importance, the search strategy to be used for identification of trials, and the methods of quality assessment, data extraction, and qualitative and quantitative synthesis of results.

Identification of trials
A broad search of the literature was conducted jointly by a research fellow experienced in literature searching and a nephrologist. The intention was to identify as many RCTs as possible relevant to the management of ESRD. By conducting a single broad search, rather than separate searches for each topic, we avoided duplication through overlap. In addition, this strategy facilitated establishment of a register of RCTs in the management of ESRD for use when undertaking further systematic reviews in this area. The steps undertaken to identify all relevant RCTs are outlined below.

Searching electronic bibliographic databases
Five electronic databases were searched systematically: Medline (National Library of Medicine database, US) on Ovid, CD plus; Embase (Elsevier Science Publishers BV, Amsterdam) on BIDS; CINAHL (Cumulative Index of Nursing and Allied Health) on Ovid, CD plus, Biosis (Biological abstracts) on Silver Platter; and the Cochrane Library database. Medline was searched using the search strategy developed by the Cochrane Collaboration for RCTs in Medline (on Silver Platter) [5]. This strategy was combined with additional search terms describing the management of ESRD and can be obtained from the authors.

The Medline search strategy was modified for searching other databases. The modifications involved changing the syntax to suit that of the search software supplier and interrogating the thesaurus or indices of each database to find equivalents of the MeSH (Medical Subject Headings) used in Medline (i.e. the database-controlled language).

All electronically derived abstracts and study titles were read by one of a team of three subject experts (nephrologists). Each abstract was assessed using a standard form, to identify abstracts that were judged as possible RCTs or quasi-RCTs (those where allocation was made according to date of birth or other less secure means of randomization) related to the management of ESRD.

Hand searching
Full-text searching of Kidney International (including supplements) from January 1985 to December 1996 was performed. Each copy (excluding conference abstracts) was searched page-by-page to identify any possible randomized or quasi-randomized controlled trials by a nephrologist and a methodologist. In keeping with the philosophy and practice of the Cochrane Collaboration, all such trials, regardless of whether relevant to the management of ESRD or not, were flagged, so that these copies of Kidney International require no future handsearching. All identified trials have been notified to the Cochrane Renal Group and to the Baltimore Cochrane Center (the co-ordinating centre for hand searching within the Cochrane Collaboration) and will form part of the International Register of RCTs.

Other methods of ascertainment of RCTs
Other methods of ascertainment of relevant RCTs included:

(i) Bibliographic list searching. The bibliographic listings of narrative reviews, identified RCTs, and relevant book chapters were checked for possible RCTs. We limited our bibliography searching to the ‘first generation’ studies only, that is we did not search the bibliographies of studies identified from a previous bibliography search.
(ii) Asking other experts in the field. The authors of trials included in each systematic review and those from whom clarification of methodology had been sought were contacted to ascertain if they knew of any other possible relevant RCTs.
(iii) Seeking information from biomedical companies. Where relevant, biomedical companies were contacted for details of any other relevant possible RCTs, published or unpublished.

Register of RCTs
All possible RCTs were imported or entered into a reference managing package (Reference Manager, Version 6.01 N). Keywords were added to identify those relevant to the chosen topics.

Quality assessment of full studies
Hard copies of possible RCTs were retrieved and were assessed for subject relevance, eligibility and methodological quality independently by a nephrologist and a research methodologist using a standard form (available from the authors on request). Quality of description and details of random allocation, blinding, withdrawals, drop-outs, numbers lost to follow-up and whether intention-to-treat analysis was possible were recorded. The assessors were not blinded to author, institution or journal. Any differences of opinion were resolved by discussion. If agreement could not be reached, a second double-assessor team assessed the study, aware of the previous team's deliberations, and had to reach a final decision.

Data abstraction
Specific data abstraction forms were generated for each review prior to the abstraction of the actual data from each study. In this way, only data on outcomes which had been specified as important in advance were abstracted from each study.

Data synthesis
When results from the individual studies could be combined quantitatively, a statistical meta-analysis of the data was undertaken. The aim of the meta-analysis was to determine the likelihood that the pattern of results reflected chance differences between the groups, and to estimate a ‘typical’ effect size of the intervention. A ‘typical’ odds ratio was calculated for binary outcomes, and a weighted mean difference was calculated for continuous outcomes. Data synthesis was carried out using the customised software Review Manager, which is available for the preparation, synthesis, and statistical analysis of systematic reviews.

Economic evaluation and search for economic studies
To obtain estimates of cost, the included RCTs were also critically assessed by a health economist to identify information on resource use that could then be combined with information on the price of those resources to provide an estimate of cost. It was anticipated before undertaking the systematic reviews, and subsequently confirmed by experience, that RCTs would not provide sufficient information on resource use. An additional search strategy was therefore, employed which followed the methodology described above but omitted search terms which limited the search to RCTs and included search terms relevant to economics. The data on outcomes (obtained from the RCTs) and on resource use could then be combined [6].



   Results
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
The systematic searches of the five electronic databases generated nearly 12 000 potentially relevant abstracts for assessment (Figure 1Go). Of these, 2085 (18%) were deemed to be possible RCTs or quasi-RCTs relevant to the management of ESRD after the citation/abstract had been assessed, and 340 (3% of total studies identified) were relevant to our six specific topics. Full hard-copy assessment of these was undertaken, which resulted in the final inclusion of 39 trials (0.3% of the total abstracts) in our reviews. Other methods of ascertainment of RCTs or quasi-RCTs (hand-searching, bibliography-checking, and contacting authors) contributed an extra nine studies (Table 2Go). The electronic search of Medline provided the majority (71%) of the included studies (Table 3Go). Despite nearly 12 000 abstracts being assessed, however, no RCTs of CAPD vs haemodialysis were identified.



View larger version (48K):
[in this window]
[in a new window]
 
Fig. 1. Summary of electronic search for RCTs and quasi-RCTs. 1Not RCT/quasi RCT, or not relevant to the precise six questions of the reviews.

 

View this table:
[in this window]
[in a new window]
 
Table 2. Ascertainment of trials from other sources

 

View this table:
[in this window]
[in a new window]
 
Table 3. Source of RCTs summarized by topic

 
In the separate electronic search for economic studies, 4168 studies were identified with 820 (20%) deemed relevant to the economic aspects of our six topics after assessment of citations/abstracts (Table 4Go). Again, although 529 studies relevant to the economics of CAPD vs haemodialysis and 109 relevant to the economics of haemodialysis duration and frequency were identified, no RCTs for the former and only one relevant to the latter were identified.


View this table:
[in this window]
[in a new window]
 
Table 4. Results of searches for economic studies

 
The full results and conclusions of the six individual reviews will be reported separately in other papers.



   Discussion
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
The management of ESRD by dialysis is one of the most resource-intensive areas of health care and the demand to establish the most effective and efficient treatment options is increasing. The considerable variation in practice between renal units in Europe further illustrates the need for evidence-based practice. This was recognized by the UK NHS R&D Health Technology Assessment Programme, which identified the management of ESRD by dialysis as one of its first priorities for research.

Systematic reviews attempt to strengthen the evidence base for clinical practice and establish whether research findings from individual studies are consistent and generalizable, or whether they vary significantly for different subsets of patients [7].

Dialysis therapy for ESRD has not previously been subject to systematic reviews in the manner described here, and we want others to learn from our experience. Firstly, the broad literature search and the hand searching of Kidney International were resource-intensive and exacting exercises (the register of possible RCTs identified should, however, save others considerable effort in carrying out further systematic reviews in this area). Secondly, six separate searches for each topic would, cumulatively, have been likely to have identified even greater numbers of abstracts with many duplications. Thirdly and most critically, the indexing of the present electronic databases is poor both in respect to subject and methodology, and too restrictive a search would undoubtedly have missed many possible RCTs and undermined the validity of the entire exercise. A possible limitation of our search strategy was that we did not systematically search conference abstracts. The primary reason for this decision was the size and resource consequences of undertaking this task. Furthermore, the gains in terms of additional trials, which could positively have been classified as RCTs and would not have been identified from any other source, are questionable.

This sort of exercise will become easier in the future. The Cochrane Collaboration is performing broad searches of all the principal electronic databases to identify all RCTs irrespective of subject area [8]. All RCTs identified by these broad searches, and by searches such as ours, will be listed in the newly established CENTRAL register within the Cochrane Library; hence it should only be necessary for future review groups to search CENTRAL for trials relevant to their subject [8].

Although thousands of study reports were initially identified, only a small number were included in any of the final reviews. If the analogy of a pyramid of research studies is used, then it is only the tip of that pyramid that is relevant to any particular review, yet it is the meticulous assessment of all the studies that form the base which gives the pyramid its structural validity and integrity. In addition, in contrast to some other areas, such as cancer and heart disease, clinical research in ESRD has not been characterized by large collaborative randomized trials, and the 48 RCTs finally included were, therefore, as large a number as initially hoped for. Most of these were small or inadequate for other reasons, demonstrating the need for such collaborative trials in ESRD. It was particularly disappointing that despite a comprehensive search, not a single RCT comparing CAPD with haemodialysis was identified. There is, therefore, no robust evidence to inform one of the primary decisions patients with ESRD and their nephrologists must make.

Our multi-disciplinary team included the key disciplines of clinical nephrology, health services research, health economics, statistics, and information science with their different skills and perspectives. This has led, we believe, to a scientifically more rigorous approach to the research, and also to research which will have wider applicability within the health service. Furthermore the transparent methodology makes the recommendations more likely to be accepted by those funding health care as well as those providing it.

Potential biases in the process by which the data from comparable trials are brought together and synthesized were minimized by the use of a study protocol, and standardized quality assessment and data abstraction forms. Identification of trials was systematic and described in our protocols (and hence repeatable). As far as possible, all relevant trials were considered, including those from less well-known sources. The methodological quality of the studies was assessed objectively to gauge the extent of any possible bias and to check for consistency between the most sound and the less rigorously conducted trials. Data were abstracted systematically to derive a common dataset for all trials; the data to be abstracted were identified before obtaining the trial reports, thus reducing the chances of concluding that chance differences were real. In addition, because results from similar trials conducted in different places amongst somewhat different groups of patients were considered, the reviews allow more confident generalization of findings to a broad range of patients.

Although it is important to determine which interventions are most effective, it is increasingly being recognized that this must be balanced against the resource consequences. As yet, however, there is no standardized format for inclusion of such economic evaluations within the structure of systematic reviews of clinical effectiveness such as those conducted under the auspices of the Cochrane Collaboration. Our reviews are the first to attempt to do this.

The main purpose of such reviews is to estimate the benefit in terms of patient outcomes and cost. This will allow judgements to be made as to the scale of the provision of particular interventions in the light of the inevitably limited resources available. The primary aim of adopting such a rigorous methodology is to reduce bias and improve the validity of recommendations from reviews which can then be incorporated into relevant sections of clinical standards or guidelines.

We have demonstrated that the methodology promoted by the Cochrane Renal Group is feasible, and through the development of a register of RCTs for ESRD, this form of research should become easier in the future. The results of this type of research have the potential to provide information of crucial importance to clinicians involved in the management of patients with renal disease, and we believe that it deserves the support of the nephrology community.



   Acknowledgments
 
We would like to thank Carol Ritchie for clerical assistance, Ian Ronald for assistance in the processing of the electronic searches, Karla Fitzhugh for data extraction, and Gloria Montague for secretarial assistance. This study was funded by the UK NHS R&D Health Technology Assessment Programme. The Health Services and Health Economics Research Units are funded by the Chief Scientist Office of the Scottish Executive Health Department. Cam Donaldson is Svare Chair in Health Economics and Alberta Heritage Senior Scholar at the University of Calgary. The views expressed are not necessarily those of the funding bodies.



   Notes
 
Correspondence and offprint requests to: Marion K. Campbell, Health Services Research Unit, University of Aberdeen, Polwarth Building, Foresterhill, Aberdeen AB25 2ZD, UK. Back



   References
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 

  1. Pocock SJ. Clinical Trials: a Practical Approach. Wiley, Chichester, 1983
  2. Sibbald B, Roland M. Understanding controlled trials: why are randomised controlled trials important? BMJ1998; 316: 201[Free Full Text]
  3. Fouque D, Laville M, Haugh M, Boissel JP. Systematic reviews and their roles in promoting evidence-based medicine in renal disease. Nephrol Dial Transplant1996; 11: 2398–2401[ISI][Medline]
  4. Cochrane Collaboration Handbook. In: The Cochrane Library (database on disk and CDROM). The Cochrane Collaboration, Oxford: Update Software; 1996—updated quarterly
  5. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ1994; 309: 1286–1291[Abstract/Free Full Text]
  6. Vale L, Donaldson C, Daly C et al. Evidence based medicine and health economics: a case study of end-stage renal disease. HERU discussion paper, 03/98, Health Economics Research Unit, University of Aberdeen, 1998
  7. Mulrow CD. Rationale for systematic reviews. BMJ1994; 309: 597–599[Free Full Text]
  8. The Cochrane Library (database on disk and CDROM). The Cochrane Collaboration, Oxford: Update Software; 1996– updated quarterly
Received for publication: 18. 1.99
Revision received 2. 8.00.