1 Health Services Research Unit, 2 Department of Medicine and Therapeutics, 3 Health Economics Research Unit, 4 Medical School Library, University of Aberdeen, Aberdeen, UK, 5 Departments of Economics and Community Health Sciences, University of Calgary, Canada
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods. Our methodology was based on that recommended by the Cochrane Collaboration (an international initiative set up to perform and disseminate systematic reviews of health care). It involved a systematic search of electronic databases and bibliographic reference lists, together with handsearching of Kidney International for studies relevant to the management of ESRD, followed by a systematic assessment of study quality.
Results. Around 12 000 abstracts were assessed which had been identified from electronic sources. Of these, 2085 (18%) were deemed to be reports of possible randomized or quasi-randomized controlled trials relevant to the management of ESRD. Three hundred and forty were relevant to the six specific reviews, and after assessment of the full manuscripts, 39 studies were finally included in our reviews. Reports of a further nine trials, which were identified from other sources, were also included. The broad search adopted allowed the parallel development of a register of trials of all aspects of the management of ESRD.
Conclusions. This study has demonstrated that the methodology of systematic reviews, as promoted by the Cochrane Renal Group, is feasible but has significant resource implications. The development of a register of randomized controlled trials (RCTs) related to the management of ESRD will facilitate this form of research in the future.
Keywords: end-stage renal disease; randomized controlled trials; systematic reviews
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The randomized controlled trial (RCT) is now widely accepted as the gold standard method for assessing the relative effectiveness of alternative treatments [1,2]. This approach to evaluation ensures that selection and other common sources of bias are minimized. The interpretation of the results of individual trials may, however, be difficult, as few are of sufficient size to rule out chance as an explanation for observed differences between treatments. In addition, entry criteria for inclusion of patients into individual trials may be too restrictive to allow confident extrapolation of the results to a particular patient. Considering all relevant trials together should redress the limitations of individual trials.
Literature reviews can, however, potentially introduce other sources of bias such as through failure to identify all trials or to take into account methodological quality. Within what has become known as a systematic review, all relevant trials are identified in a systematic and reproducible manner, the methodological quality of individual trials is assessed objectively, and data are abstracted and synthesized according to a common protocol.
Whilst a systematic review of all relevant trials will address which treatment option is more effective, it is also important to assess the impact on resource use and hence efficiency. If, for example, one method of delivering care to patients with ESRD is shown to be more effective and less costly than its alternative, it should then be recommended as the treatment of choice. If, however, the more effective option is more costly, the magnitude of this extra cost may help to decide whether it is justified in terms of the improvement in patient outcome.
This paper describes the methodology used to conduct six systematic reviews of both the effectiveness and efficiency of aspects of the management of ESRD (excluding transplantation) commissioned by the UK NHS Research and Development Health Technology Assessment Programme. These are some of the first systematic reviews to be undertaken in ESRD and will contribute to the work of the Cochrane Renal Group [3].
![]() |
Methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
Protocols
A protocol describing the proposed process was formulated for each topic which explicitly described the objectives of the review, the clinical and methodological criteria required of studies for inclusion, the outcome measures of importance, the search strategy to be used for identification of trials, and the methods of quality assessment, data extraction, and qualitative and quantitative synthesis of results.
Identification of trials
A broad search of the literature was conducted jointly by a research fellow experienced in literature searching and a nephrologist. The intention was to identify as many RCTs as possible relevant to the management of ESRD. By conducting a single broad search, rather than separate searches for each topic, we avoided duplication through overlap. In addition, this strategy facilitated establishment of a register of RCTs in the management of ESRD for use when undertaking further systematic reviews in this area. The steps undertaken to identify all relevant RCTs are outlined below.
Searching electronic bibliographic databases
Five electronic databases were searched systematically: Medline (National Library of Medicine database, US) on Ovid, CD plus; Embase (Elsevier Science Publishers BV, Amsterdam) on BIDS; CINAHL (Cumulative Index of Nursing and Allied Health) on Ovid, CD plus, Biosis (Biological abstracts) on Silver Platter; and the Cochrane Library database. Medline was searched using the search strategy developed by the Cochrane Collaboration for RCTs in Medline (on Silver Platter) [5]. This strategy was combined with additional search terms describing the management of ESRD and can be obtained from the authors.
The Medline search strategy was modified for searching other databases. The modifications involved changing the syntax to suit that of the search software supplier and interrogating the thesaurus or indices of each database to find equivalents of the MeSH (Medical Subject Headings) used in Medline (i.e. the database-controlled language).
All electronically derived abstracts and study titles were read by one of a team of three subject experts (nephrologists). Each abstract was assessed using a standard form, to identify abstracts that were judged as possible RCTs or quasi-RCTs (those where allocation was made according to date of birth or other less secure means of randomization) related to the management of ESRD.
Hand searching
Full-text searching of Kidney International (including supplements) from January 1985 to December 1996 was performed. Each copy (excluding conference abstracts) was searched page-by-page to identify any possible randomized or quasi-randomized controlled trials by a nephrologist and a methodologist. In keeping with the philosophy and practice of the Cochrane Collaboration, all such trials, regardless of whether relevant to the management of ESRD or not, were flagged, so that these copies of Kidney International require no future handsearching. All identified trials have been notified to the Cochrane Renal Group and to the Baltimore Cochrane Center (the co-ordinating centre for hand searching within the Cochrane Collaboration) and will form part of the International Register of RCTs.
Other methods of ascertainment of RCTs
Other methods of ascertainment of relevant RCTs included:
Register of RCTs
All possible RCTs were imported or entered into a reference managing package (Reference Manager, Version 6.01 N). Keywords were added to identify those relevant to the chosen topics.
Quality assessment of full studies
Hard copies of possible RCTs were retrieved and were assessed for subject relevance, eligibility and methodological quality independently by a nephrologist and a research methodologist using a standard form (available from the authors on request). Quality of description and details of random allocation, blinding, withdrawals, drop-outs, numbers lost to follow-up and whether intention-to-treat analysis was possible were recorded. The assessors were not blinded to author, institution or journal. Any differences of opinion were resolved by discussion. If agreement could not be reached, a second double-assessor team assessed the study, aware of the previous team's deliberations, and had to reach a final decision.
Data abstraction
Specific data abstraction forms were generated for each review prior to the abstraction of the actual data from each study. In this way, only data on outcomes which had been specified as important in advance were abstracted from each study.
Data synthesis
When results from the individual studies could be combined quantitatively, a statistical meta-analysis of the data was undertaken. The aim of the meta-analysis was to determine the likelihood that the pattern of results reflected chance differences between the groups, and to estimate a typical effect size of the intervention. A typical odds ratio was calculated for binary outcomes, and a weighted mean difference was calculated for continuous outcomes. Data synthesis was carried out using the customised software Review Manager, which is available for the preparation, synthesis, and statistical analysis of systematic reviews.
Economic evaluation and search for economic studies
To obtain estimates of cost, the included RCTs were also critically assessed by a health economist to identify information on resource use that could then be combined with information on the price of those resources to provide an estimate of cost. It was anticipated before undertaking the systematic reviews, and subsequently confirmed by experience, that RCTs would not provide sufficient information on resource use. An additional search strategy was therefore, employed which followed the methodology described above but omitted search terms which limited the search to RCTs and included search terms relevant to economics. The data on outcomes (obtained from the RCTs) and on resource use could then be combined [6].
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
|
|
|
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Systematic reviews attempt to strengthen the evidence base for clinical practice and establish whether research findings from individual studies are consistent and generalizable, or whether they vary significantly for different subsets of patients [7].
Dialysis therapy for ESRD has not previously been subject to systematic reviews in the manner described here, and we want others to learn from our experience. Firstly, the broad literature search and the hand searching of Kidney International were resource-intensive and exacting exercises (the register of possible RCTs identified should, however, save others considerable effort in carrying out further systematic reviews in this area). Secondly, six separate searches for each topic would, cumulatively, have been likely to have identified even greater numbers of abstracts with many duplications. Thirdly and most critically, the indexing of the present electronic databases is poor both in respect to subject and methodology, and too restrictive a search would undoubtedly have missed many possible RCTs and undermined the validity of the entire exercise. A possible limitation of our search strategy was that we did not systematically search conference abstracts. The primary reason for this decision was the size and resource consequences of undertaking this task. Furthermore, the gains in terms of additional trials, which could positively have been classified as RCTs and would not have been identified from any other source, are questionable.
This sort of exercise will become easier in the future. The Cochrane Collaboration is performing broad searches of all the principal electronic databases to identify all RCTs irrespective of subject area [8]. All RCTs identified by these broad searches, and by searches such as ours, will be listed in the newly established CENTRAL register within the Cochrane Library; hence it should only be necessary for future review groups to search CENTRAL for trials relevant to their subject [8].
Although thousands of study reports were initially identified, only a small number were included in any of the final reviews. If the analogy of a pyramid of research studies is used, then it is only the tip of that pyramid that is relevant to any particular review, yet it is the meticulous assessment of all the studies that form the base which gives the pyramid its structural validity and integrity. In addition, in contrast to some other areas, such as cancer and heart disease, clinical research in ESRD has not been characterized by large collaborative randomized trials, and the 48 RCTs finally included were, therefore, as large a number as initially hoped for. Most of these were small or inadequate for other reasons, demonstrating the need for such collaborative trials in ESRD. It was particularly disappointing that despite a comprehensive search, not a single RCT comparing CAPD with haemodialysis was identified. There is, therefore, no robust evidence to inform one of the primary decisions patients with ESRD and their nephrologists must make.
Our multi-disciplinary team included the key disciplines of clinical nephrology, health services research, health economics, statistics, and information science with their different skills and perspectives. This has led, we believe, to a scientifically more rigorous approach to the research, and also to research which will have wider applicability within the health service. Furthermore the transparent methodology makes the recommendations more likely to be accepted by those funding health care as well as those providing it.
Potential biases in the process by which the data from comparable trials are brought together and synthesized were minimized by the use of a study protocol, and standardized quality assessment and data abstraction forms. Identification of trials was systematic and described in our protocols (and hence repeatable). As far as possible, all relevant trials were considered, including those from less well-known sources. The methodological quality of the studies was assessed objectively to gauge the extent of any possible bias and to check for consistency between the most sound and the less rigorously conducted trials. Data were abstracted systematically to derive a common dataset for all trials; the data to be abstracted were identified before obtaining the trial reports, thus reducing the chances of concluding that chance differences were real. In addition, because results from similar trials conducted in different places amongst somewhat different groups of patients were considered, the reviews allow more confident generalization of findings to a broad range of patients.
Although it is important to determine which interventions are most effective, it is increasingly being recognized that this must be balanced against the resource consequences. As yet, however, there is no standardized format for inclusion of such economic evaluations within the structure of systematic reviews of clinical effectiveness such as those conducted under the auspices of the Cochrane Collaboration. Our reviews are the first to attempt to do this.
The main purpose of such reviews is to estimate the benefit in terms of patient outcomes and cost. This will allow judgements to be made as to the scale of the provision of particular interventions in the light of the inevitably limited resources available. The primary aim of adopting such a rigorous methodology is to reduce bias and improve the validity of recommendations from reviews which can then be incorporated into relevant sections of clinical standards or guidelines.
We have demonstrated that the methodology promoted by the Cochrane Renal Group is feasible, and through the development of a register of RCTs for ESRD, this form of research should become easier in the future. The results of this type of research have the potential to provide information of crucial importance to clinicians involved in the management of patients with renal disease, and we believe that it deserves the support of the nephrology community.
![]() |
Acknowledgments |
---|
![]() |
Notes |
---|
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|