Predicting 1 year mortality in an outpatient haemodialysis population: a comparison of comorbidity instruments

Dana C. Miskulin1, Alice A. Martin1, Richard Brown2, Nancy E. Fink3, Josef Coresh3, Neil R. Powe3, Philip G. Zager2, Klemens B. Meyer1 and Andrew S. Levey1 and the Medical Directors of Dialysis Clinic, Inc.4

1Division of Nephrology, Tufts-New England Medical Center, Boston, MA, USA, 3Johns Hopkins Medical Institutions, Baltimore, MD, USA, 2University of New Mexico, Albuquerque, NM, USA and 4Dialysis Clinic, Inc., Nashville, TN, USA

Correspondence and offprint requests to: Dana Miskulin, MD, MS, New England Medical Center, Division of Nephrology, Box 391, 750 Washington Street, Boston, MA 02111, USA. Email: dmiskulin{at}tufts-nemc.org



   Abstract
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Background. A valid and practical measure of comorbid illness burden in dialysis populations is greatly needed to enable unbiased comparisons of clinical outcomes. We compare the discriminatory accuracy of 1 year mortality predictions derived from four comorbidity instruments in a large representative US dialysis population.

Methods. Comorbidity information was collected using the Index of Coexistent Diseases (ICED) in 1779 haemodialysis patients of a national dialysis provider between 1997 and 2000. Comorbidity was also scored according to the Charlson Comorbidity Index (CCI), Wright-Khan and Davies indices. Relationships of instrument scores with 1 year mortality were assessed in separate logistic regression analyses. Discriminatory ability was compared using the area under the receiver-operating characteristics curve (AUC), based on predictions of each regression model.

Results. When mortality was predicted using comorbidity and age, the ICED better discriminated between survivors and those who died (AUC 0.72) as compared with the CCI (0.67), Wright-Khan (0.68) and Davies (0.68) indices. Upon addition of race and serum albumin, predictive accuracy of each model improved further (AUCs of the ICED, 0.77; CCI, 0.75; Wright-Khan Index, 0.75; Davies Index, 0.74).

Conclusions. The ICED had greater discriminatory ability than the CCI, Davies and Wright-Khan indices, when age and a comorbidity index were used alone to predict 1 year mortality; however, the differences among instruments diminished once serum albumin, race and the cause of ESRD were accounted for. None of the currently available comorbidity instruments tested in this study discriminated mortality outcomes particularly well. Assessing comorbidity using the ICED takes significantly more time. Identifying the key prognostic comorbid conditions and weighting these according to outcomes in a dialysis population should increase accuracy and, with restriction to a finite number of items, provide a practical means for widespread comorbidity assessment.

Keywords: case-mix severity; comorbidity; risk stratification



   Introduction
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
The fair (unbiased) comparison of outcomes across study arms of a clinical trial or across dialysis units in a quality assessment programme requires adjustment for case-mix differences among the groups being compared [1,2]. Multiple factors determine case-mix severity, i.e. the risk of adverse outcomes, of which the burden of comorbid illness is one of the most important [14]. There is no consensus on the measurement and grading of comorbid illnesses in dialysis patients.

A single standardized method for quantifying comorbid illness burden would enable uniform case-mix adjustment. Various instruments have been used in studies involving dialysis patients, including: the Charlson Comorbidity Index (CCI), a generic index developed from a general medical inpatient population [5]; the Index of Coexistent Diseases (ICED), a generic tool modified for dialysis patients [6]; and the Davies [7] and Wright-Khan indices [8,9], both developed specifically for dialysis populations. Each has been validated for the outcome of mortality in dialysis populations, with a graded increase in mortality risk predicted per increment in instrument level [3,4,612]. The prognostic accuracy of the CCI, Wright-Khan and Davies indices for 2 year mortality was recently compared in the Netherlands Cooperative Study on Dialysis Adequacy (NECOSAD Study), an observational cohort study involving1041 incident dialysis patients from 36 centres in the Netherlands. In general, these indices capture the presence, but not the severity, of disease. A new index with explicitly defined severity levels for four conditions (diabetes, ischaemic heart disease, congestive heart failure and malignancy) was developed and tested in this study, with results showing similar discriminatory performance as the other instruments [area under the receiver-operating characteristic curves (AUCs) of 0.72–0.75] [13]. The authors concluded that the characterization of disease severity made no difference to prognostic power, yet acknowledged that the scope of definition of disease severity was limited.

A comparison with the ICED, which differs considerably from the other instruments in the detail by which comorbidity is characterized, has not been performed. In contrast to the CCI, Wright-Khan and Davies’ indices, which broadly categorize comorbid illness, the ICED is a 160 item questionnaire that explicitly characterizes disease severity, using clinically defined severity levels for each of 19 medical conditions and 11 physical impairments. With the increases in age and prevalence of comorbid illnesses noted in the US incident dialysis population over the past decade, it would be increasingly important to identify not only the presence, but also the severity of comorbid illnesses in quantifying individuals’ risk for adverse outcomes. The objective of the present study was to compare four commonly used comorbidity instruments, the ICED, the CCI, the Wright-Khan Index and the Davies Index, for their accuracy in predicting 1 year mortality in a large representative haemodialysis population.



   Subjects and methods
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Study population
A pilot programme to assess the feasibility of collecting comorbidity information using the ICED as part of routine practice was conducted in 41 dialysis units of a not-for-profit US national dialysis provider [Dialysis Clinic, Inc. (DCI)] from October 1997 to October 2000 [14]. Participation of these dialysis units was voluntary. The mean (SD) length of a single dialysis session for the DCI population overall in the year 2000 was 3.6 (0.52) h, patients were treated three times per week and 96% of treatments were performed with substituted cellulose or synthetic membranes. Forty percent of the study population had comorbidity assessed within 4 months of starting chronic outpatient dialysis therapy.

Data collection
The Medical Information System (MIS) is an electronic medical record database used by all dialysis units throughout the DCI network. Nurses and other staff at the local dialysis units routinely enter clinical information relating to the care of the patient into the MIS, including progress notes, treatment sessions, medication lists, hospitalizations and reasons for admission. Monthly laboratory results are linked directly to the MIS. A quality management committee routinely monitors data for accuracy and completeness. Deaths are recorded by the dialysis unit staff and validated against data submitted to the Centers for Medicare and Medicaid Services (CMS) on the Death Notification Form (Form 2748). Transfer out of the dialysis unit, loss-to-follow-up and modality switches are also routinely recorded within the MIS. Prospective collection of comorbidity data ended February 1, 2001, to enable assessment of the feasibility of the programme and the validity of the risk assessment. The end of observation for these analyses was June 31, 2001.

Assessment of comorbidity using the ICED
Comorbidity information was abstracted and scored according to the definitions of the ICED. The ICED consists of 166 items that characterize both the presence and severity of 19 medical conditions and 11 physical impairments. Details of the instrument and its scoring are provided elsewhere [15]. In brief, it consists of two assessments: the Index of Disease Severity (IDS), based on data abstracted from the medical records, and an assessment of physical impairments [Index of Physical Impairments (IPI)], based on the dialysis nurses’ observation of the patient in the outpatient dialysis unit. To score the IDS, nurses were instructed to update and review discharge summaries, problem lists (routinely maintained in DCI patients), consultation letters, nurse and physician progress notes, medication lists, diagnostic imaging reports, and the nephrologists’ history and physical exam(s). The final ICED score is based on an algorithm, which combines the single peak disease category (IDS) with the single peak physical impairment category (IPI), ranging from 0–3 with 3 as the most severe. ICED 0 was combined with ICED 1 because the former was infrequent (<0.5%).

The CCI, Wright-Khan and Davies indices were scored retrospectively from the comorbidity database assembled above. The ICED includes all items within the other instruments for all conditions except malignancy, the latter is defined in more detail in the CCI. Thus, for these analyses we excluded subjects with a malignancy within the past 5 years (7% of the population). Item definitions and weights used were those defined for the original instruments [3,5,8,9]. For the CCI, the weighted items were summed and divided into levels as defined by Fried et al. [11]. We were able to define an additional higher severity level than the prior study due to our larger sample size and/or the greater comorbidity severity of our population. The time taken to review the medical record and complete the comorbidity assessment using each instrument was not recorded in this study; however, other studies have reported an average of 50 min per patient with the ICED [15,16] and 20 min with the CCI [11]. Completion time for the Wright-Khan and Davies indices has not been reported in prior studies, but is likely to be similar or less than for the CCI.

Statistical methods
The outcome of interest was death within 1 year of the ICED assessment. The start of the observation time was the date of the ICED assessment. Patients were censored at transplant, transfer to a non-participating clinic, or June 1, 2001, whichever came first. No patients were reported to recover kidney function. Eight patients were reported to have withdrawn from dialysis, although a death date was not reported despite more than 3 months follow-up after the censoring date in each case. These events may have been a recovery of renal function or a transfer from the unit. As a death was not recorded, the date of last observation was used for these eight subjects and a death was not counted. Baseline characteristics were described using means and SDs for continuous variables and frequencies for categorical variables. Differences across ICED strata (0–1, 2, 3) were tested for significance using one-way ANOVA for continuous variables and chi-squared tests for categorical variables.

Univariate associations of each instrument with 1 year mortality were assessed in separate logistic regression models with 1 year mortality as the binary outcome. Logistic regression was used as it requires fewer assumptions than Cox proportional hazards regression and the output of the model is readily translated into a 1 year probability of death. The CCI and Wright-Khan Index incorporate age in the scoring, but both instruments were noted to perform better when age was added as a separate covariate in a previous study [13], thus, age was added to models in the present study. Adjusted multivariable models were constructed through the sequential addition of vintage (defined as <1 year, 1–3 years and >3 years since the start of dialysis), race (white, black and other), the cause of ESRD (glomerulonephritis, hypertension, diabetes, other, polycystic kidney disease), and serum albumin. Serum albumin was averaged over 30 days prior and 60 days subsequent to the date of comorbidity data collection. The change in the fit of the model upon addition of each covariate was expressed as the change in the –2 log-likelihood test statistic over the prior model. The overall fit of the model is expressed as the change in the –2 log-likelihood test statistic over the null model. The interaction of incident status (<=4 months vs >4 months since start of dialysis) was tested with each comorbidity instrument to determine whether instruments performed differently in incident vs prevalent populations.

Assessing model discrimination and calibration
One year mortality was predicted from univariate and multivariable-adjusted models derived using each instrument. Predictive performance of each instrument was quantified as the area under the receiver-operating characteristic curves (AUC) [17,18]. The receiver-operating characteristic curve (ROC) plots the false-positive rate (x-axis) as the cut-off for a true-positive result (y-axis) is varied. An AUC of 1.0 indicates that the instrument correctly orders all possible pairs of patients, given one who dies and one who survives, with a higher predicted risk of death to the patient who dies, while an AUC of 0.50 suggests the instrument is no better than chance alone. Ninety-five percent confidence intervals (CIs) were estimated for AUCs with the assumption of non-parametric distribution [19]. To exclude the presence of bias at the lower or upper end of the risk spectrum, the proportions of observed vs expected deaths were compared across deciles of risk using the Hosmer-Lememshow chi-squared test [20]. Data were analysed using SPSS Version 10.0 (Chicago, IL).



   Results
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Exclusions
Comorbidity was assessed at least once in 2400 patients, ~40%, within 4 months of the patient's initiation of outpatient dialysis. One hundred and sixty-one peritoneal dialysis patients were excluded as their numbers were small relative to haemodialysis and statistical power was felt to be limited for assessing differential effects in this subgroup independently. By the definition of the outcome of 1 year mortality, 460 patients with <1 year of follow-up were excluded. This left a sample of 1779 for comparison of instruments. Excluded patients were more likely to be incident, but were otherwise similar to those retained in the analytic data set. There were 434 deaths (24.4%) within 1 year of comorbidity assessment.

Baseline characteristics stratified by comorbid illness severity
The study population used in the analyses was representative of the US dialysis population: the combined mean age was 62 years, 47% were female, 30% were African–American, and 44% had diabetes as the cause of ESRD. Laboratory values averaged over a 90 day window of the comorbidity assessment were as follows: the mean (SD) serum albumin was 3.70 (0.44) g/dl, the mean (SD) haematocrit was 34.5% (3.6) and 83% had a Kt/V >=1.2. Approximately 40% of the patients had comorbidity assessments performed within 1 year of starting dialysis. With increasing comorbidity severity, as measured by the ICED, patients were older, a greater proportion were Caucasian, and had diabetes as the cause of ESRD (Table 1). Increasing comorbidity severity was also significantly correlated with decreasing serum albumin (r = –0.21, P<0.0001).


View this table:
[in this window]
[in a new window]
 
Table 1. Baseline characteristics stratified by comorbidity score

 
Relationships of comorbidity instruments with 1 year mortality
The mortality rate increases with each increment in instrument level, as per the second column of Table 2. The fit of the model with the ICED (and age) was significantly greater than for the other instruments (and age), as reflected in the –2 log-likelihood test statistic. Discriminatory ability (i.e. the ability of the model to assign a higher score to an individual who died, given all possible pairs in which one individual dies and one survives), was highest for the ICED, with an AUC (95% CI) of 0.72 (0.69–0.75) vs 0.67 (0.65–0.70) for the CCI, and 0.68 (0.65–0.70) for the Wright-Khan and Davies indices each. The 95% CIs among instruments overlapped slightly. All instruments were calibrated, indicating that there were no differences in observed vs predicted outcomes at high vs low ends of the spectrum of predicted risk.


View this table:
[in this window]
[in a new window]
 
Table 2. Predicting 1 year mortality: a comparison of comorbidity indices and age

 
Predicting 1 year mortality using comorbidity and other case-mix factors
Upon successive addition of covariates other than comorbidity, mortality predictions improved, as reflected in the increase in AUCs in Table 3. The model fit was not improved with adjustment for serum phosphate, calcium-phosphate product, haematocrit, creatinine, Kt/V or body mass index (the latter was missing in 171 subjects). The discriminatory performance of the four adjusted models was no longer significantly different (AUCs were as follows: ICED, 0.77; CCI, 0.75; Wright-Khan Index, 0.75; Davies Index, 0.74), yet the fit of the adjusted ICED model was better than the others, again reflected in the –2 log-likelihood test statistic. Serum albumin contributed more to the mortality prediction than the respective comorbidity instrument, in all models except that composed of the ICED. The adjusted models were calibrated across risk levels (right-hand column of Table 3), except the Davies Index, which exhibited separation of observed and predicted values (P = 0.06). Differential effects (i.e. interactions) among incident vs prevalent patients for the 1 year mortality outcome were not found for any of the comorbidity instruments.


View this table:
[in this window]
[in a new window]
 
Table 3. Contribution to 1 year mortality with successive entry of covariates into the predictive model

 


   Discussion
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
We compared 1 year mortality predictions derived from four comorbidity assessment instruments previously validated for mortality in dialysis populations. When mortality was predicted using a comorbidity instrument and age, the ICED had greater discriminatory ability than the other instruments. Upon addition of age, race and serum albumin, the prognostic accuracy of the models increased considerably, identifying the value of these routinely available factors in defining individuals’ risk. Overall predictive accuracy was, however, low—pairs of patients were misclassified (i.e. the patient who survived was incorrectly assigned a higher predicted risk than the one who died or vice versa) ~25% of the time. These instruments are not sufficiently accurate to be used solely in clinical decision-making. While these results might suggest the assessment of comorbidity adds little to risk stratification, the low predictive accuracy more likely reflects the difficulty of predicting 1 year mortality and the inadequacies of these comorbidity instruments in risk stratifying for this outcome. These results suggest that better characterization of factors associated with mortality will be necessary for accurate predictions of mortality over 1–2 years.

The ICED, which had greater discriminatory ability than the other instruments considered on their own, provides direction for improving predictive accuracy for this outcome. The key difference of the ICED from the other instruments is the detail by which disease severity is classified, as illustrated with the example of congestive heart failure in Table 4. Given the high prevalence of many comorbid conditions [16] it is important to characterize severity, and not merely the presence of disease in identifying those at highest risk. Another unique component of the ICED is the assessment of physical limitations, which has repeatedly been shown to be a strong prognostic factor in dialysis patients [21].


View this table:
[in this window]
[in a new window]
 
Table 4. Comparison of comorbidity instruments’ structure, item definitions and scoring

 
There are several important differences between generic and disease-specific instruments, which are illustrated in the instruments used in this study. A relatively large number of items of the CCI relate to comorbid conditions that are uncommon in a dialysis population, reflecting its development from a general medical inpatient population. Of the 19 comorbid conditions of the CCI, seven refer to conditions that affected <8% of this dialysis population: four for cancer (7% of this study population), one for severe liver disease (0.1%), one for AIDS (0.1%) and one for paraplegia (0.1%). While it is important to assign a high risk to these individuals, identifying these patients does not effectively stratify the majority of a dialysis population. Conversely, there are only two items relating to ischaemic heart disease and congestive heart failure, despite the high prevalence and strong prognostic significance of these comorbid illnesses. Also, the definitions of items in a generic instrument may lose meaning and/or prognostic significance when applied to a disease-specific population. For example, while the diagnosis of congestive heart failure through physical examination or the response to diuretics may identify a subset of patients with poor prognosis in the general population (as per the CCI, Table 4), it may have a different meaning (and may need to be assigned a different weight) in dialysis patients, in whom fluid overload is relatively common and does not necessarily indicate left ventricular dysfunction. A final point that applies to both the ICED and CCI, the relative weights assigned in the general population may not reflect relationships of the condition with outcomes in a dialysis population.

The reliability, or consistency in scoring across reviewers, is as important as instrument validity, given that these instruments will be used in multicentre settings. The ICED is the only instrument of the four that has been tested and found to have good reliability when used in a multicentre setting [15]. The Wright-Khan and Davies indices have been used for the most part in single centre studies, and were scored by physicians. Agreement may be very different with more reviewers, as in a multicentre study, or with reviewers of different training or background, for example, nurses compared with physicians.

Despite the advantages of a detailed and comprehensive comorbidity review, we acknowledge that the 160 item ICED may be too burdensome to be practical for everyday use in busy dialysis units. We believe comorbid conditions of key prognostic significance must be identified, defined using specific criteria and weighted according to relationships with clinical outcomes in a contemporary dialysis population, to improve upon results seen in this study.

Finally, this study has focused on the validity of comorbidity instruments for predicting 1 year mortality, yet other clinical outcomes are important endpoints of clinical trials or cost-effectiveness analyses. For example, patients’ underlying comorbid illness burden also impacts on quality of life, independence in daily living and hospitalization use. Different conditions and weights may be needed to assess relationships of individual comorbidity items with these other outcomes, which deserve to be studied.

These analyses have some limitations. The agreement between a nurse and a physician reviewer was not assessed in this study, although nurses were provided with a training manual and a resource person (D.C.M., A.A.M.) was available to field questions related to comorbidity scoring throughout the course of the study. This is similar to past studies where good inter-rater reliability in scoring the ICED was found [15,16]. The exclusion of patients with malignancy (7% of the population) might reduce predictive performance of the CCI, relative to the ICED, as it was the single comorbid condition that was more detailed in its definition than the ICED. The exclusion of this small proportion of patients, most of whom had the lowest severity weighting for this category, would be unlikely to alter performance significantly. In addition, the definitions of the ICED were used to score the CCI, Wright-Khan and Davies indices. Because the ICED has at least as much detail as these other instruments, the scoring of other instruments from this database is likely to be more accurate than the results of a prior study [13]. We cannot, however, exclude the possibility that comorbidity scoring may have differed had the respective instruments been used to abstract data from the medical record. The contribution of novel inflammatory markers such as C-reactive protein or Il-6, that have been shown to be prognostic for outcomes, were not collected and could not be compared for their contribution to the models.

In comparing comorbidity instruments used in dialysis patients over the last decade, we find low predictive accuracy for 1 year mortality. Of the instruments assessed in this study, the ICED was more accurate than the others, but this advantage was no longer present after addition of routinely available factors including, age, the cause of ESRD, race, vintage and serum albumin. None of these instruments, either alone or in combination with other factors were sufficiently accurate to be used solely in clinical decision-making. The combination of comorbidity and other factors that define case-mix severity must be accounted for in comparing outcomes in clinical trials and quality assessment programmes. Further research to identify, define and weight the key comorbidity variables of prognostic significance in a dialysis population is needed to improve the accuracy of the risk assessment.



   Acknowledgments
 
The authors express their gratitude to the dialysis staff from the 41 dialysis units of DCI who collected comorbidity data and to the patients who allowed their medical records to be reviewed for this study. Drs Miskulin, Zager and Meyer receive salary support from DCI. Presented in part at the 34th Annual Meeting of the American Society of Nephrology held in San Francisco, CA, 10–17 October, 2001.

Conflict of interest statement. None declared.



   References
 Top
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 

  1. Chertow GM. Leveling the ‘paying’ field in end-stage renal disease. Am J Med 2000; 108: 666–668[CrossRef][ISI][Medline]
  2. Iezzoni L. Risk Adjustment for Measuring Health Outcomes, 2nd edn. Health Administration Press, Ann Arbor, MI, 1997; 609 pp.
  3. Davies S, Phillips L, Naish P, Russell G. Quantifying comorbidity in peritoneal dialysis patients and its relationship to other predictors of survival. Nephrol Dial Transplant 2002; 17: 1085–1092[Abstract/Free Full Text]
  4. Miskulin D, Meyer K, Martin A et al. Baseline comorbidity and its change predict survival in an incident dialysis population. Am J Kidney Dis 2003; 41: 149–161[CrossRef][ISI][Medline]
  5. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987; 40: 373–383[ISI][Medline]
  6. Athienites NV, Miskulin DC, Fernandez GF et al. Comorbidity assessment in hemodialysis and peritoneal dialysis using the Index of Coexistent Disease (ICED). Semin Dial 2000; 13: 320–326[CrossRef][ISI][Medline]
  7. Davies S, Russell L, Bryan J, Phillips L, Russell G. Comorbidity, urea kinetics and appetite in continuous ambulatory peritoneal dialysis patients: their interrelationship and prediction of survival. Am J Kidney Dis 1995; 26: 353–361[ISI][Medline]
  8. Khan I, Graeme R, Edward N, Fleming L, Henderson I, MacLeod A. Influence of coexisting disease on survival on renal-replacement therapy. Lancet 1993; 341: 415–418[ISI][Medline]
  9. Wright L. Survival in patients with end-stage renal disease. Am J Kidney Dis 1991; 17: 25–28[ISI][Medline]
  10. Beddhu S, Bruns FJ, Saul M, Seddon P, Zeidel ML. A simple comorbidity scale predicts clinical outcomes and costs in dialysis patients. Am J Med 2000; 108: 609–613[CrossRef][ISI][Medline]
  11. Fried L, Bernardini J, Piraino B. Charlson Comorbidity Index as a predictor of outcomes in incident peritoneal dialysis patients. Am J Kidney Dis 2001; 37: 337–342[ISI][Medline]
  12. Nicolucci A, Cubasso D, Labbrozzi D et al. Effect of coexistent diseases on survival of patients undergoing dialysis. ASAIO J 1995; 41: M291–M295[Medline]
  13. van Manen J, Korevaar J, Dekker F, Boeschoten E, Bossuyt P, Krediet R. How to adjust for comorbidity in survival studies in ESRD patients: a comparison of different indices. Am J Kidney Dis 2002; 40: 82–89[CrossRef][ISI][Medline]
  14. Levey AS, Martin AA, Miskulin DC, Athienites NV, Meyer KB. Comorbidity assessment in ESRD: methods and results in a National Dialysis Chain. J Am Soc Nephrol 1999; 10: 171
  15. Miskulin D, Athienites N, Yan G et al. Comorbidity assessment using the Index of Coexistent Diseases (ICED) in a multicenter clinical trial: the Hemodialysis (HEMO) Study. Kidney Int 2001; 60: 1498–1510[CrossRef][ISI][Medline]
  16. Miskulin DC, Meyer KB, Athienites NV et al. Comorbidity and other factors associated with modality selection in incident dialysis patients: the CHOICE Study. Am J Kidney Dis 2002; 39: 324–336[ISI][Medline]
  17. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1983; 143: 29–36[ISI]
  18. Centor R, Schwartz J. An evaluation of methods for assessing the area under the receiver operating characteristic (ROC) curve. Med Decis Making 1985; 5: 149–156[Medline]
  19. Receiver Operator Characteristic Curve. SPSS 7.5 Statistical Algorithms, Chicago, IL, SPSS Inc., 1997
  20. Lememshow S, Hosmer D. A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol 1982; 115: 92–106[Abstract]
  21. McClellan W, Anson D, Birkeli K, Tuttle E. Functional status and quality of life: predictors of early mortality among patients entering treatment for end-stage renal disease. J Clin Epidemiol 1991; 44: 83–89[ISI][Medline]
Received for publication: 26. 2.03
Accepted in revised form: 24. 9.03