1Division of Nephrology, Tufts-New England Medical Center, Boston, MA, USA, 3Johns Hopkins Medical Institutions, Baltimore, MD, USA, 2University of New Mexico, Albuquerque, NM, USA and 4Dialysis Clinic, Inc., Nashville, TN, USA
Correspondence and offprint requests to: Dana Miskulin, MD, MS, New England Medical Center, Division of Nephrology, Box 391, 750 Washington Street, Boston, MA 02111, USA. Email: dmiskulin{at}tufts-nemc.org
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods. Comorbidity information was collected using the Index of Coexistent Diseases (ICED) in 1779 haemodialysis patients of a national dialysis provider between 1997 and 2000. Comorbidity was also scored according to the Charlson Comorbidity Index (CCI), Wright-Khan and Davies indices. Relationships of instrument scores with 1 year mortality were assessed in separate logistic regression analyses. Discriminatory ability was compared using the area under the receiver-operating characteristics curve (AUC), based on predictions of each regression model.
Results. When mortality was predicted using comorbidity and age, the ICED better discriminated between survivors and those who died (AUC 0.72) as compared with the CCI (0.67), Wright-Khan (0.68) and Davies (0.68) indices. Upon addition of race and serum albumin, predictive accuracy of each model improved further (AUCs of the ICED, 0.77; CCI, 0.75; Wright-Khan Index, 0.75; Davies Index, 0.74).
Conclusions. The ICED had greater discriminatory ability than the CCI, Davies and Wright-Khan indices, when age and a comorbidity index were used alone to predict 1 year mortality; however, the differences among instruments diminished once serum albumin, race and the cause of ESRD were accounted for. None of the currently available comorbidity instruments tested in this study discriminated mortality outcomes particularly well. Assessing comorbidity using the ICED takes significantly more time. Identifying the key prognostic comorbid conditions and weighting these according to outcomes in a dialysis population should increase accuracy and, with restriction to a finite number of items, provide a practical means for widespread comorbidity assessment.
Keywords: case-mix severity; comorbidity; risk stratification
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
A single standardized method for quantifying comorbid illness burden would enable uniform case-mix adjustment. Various instruments have been used in studies involving dialysis patients, including: the Charlson Comorbidity Index (CCI), a generic index developed from a general medical inpatient population [5]; the Index of Coexistent Diseases (ICED), a generic tool modified for dialysis patients [6]; and the Davies [7] and Wright-Khan indices [8,9], both developed specifically for dialysis populations. Each has been validated for the outcome of mortality in dialysis populations, with a graded increase in mortality risk predicted per increment in instrument level [3,4,612]. The prognostic accuracy of the CCI, Wright-Khan and Davies indices for 2 year mortality was recently compared in the Netherlands Cooperative Study on Dialysis Adequacy (NECOSAD Study), an observational cohort study involving1041 incident dialysis patients from 36 centres in the Netherlands. In general, these indices capture the presence, but not the severity, of disease. A new index with explicitly defined severity levels for four conditions (diabetes, ischaemic heart disease, congestive heart failure and malignancy) was developed and tested in this study, with results showing similar discriminatory performance as the other instruments [area under the receiver-operating characteristic curves (AUCs) of 0.720.75] [13]. The authors concluded that the characterization of disease severity made no difference to prognostic power, yet acknowledged that the scope of definition of disease severity was limited.
A comparison with the ICED, which differs considerably from the other instruments in the detail by which comorbidity is characterized, has not been performed. In contrast to the CCI, Wright-Khan and Davies indices, which broadly categorize comorbid illness, the ICED is a 160 item questionnaire that explicitly characterizes disease severity, using clinically defined severity levels for each of 19 medical conditions and 11 physical impairments. With the increases in age and prevalence of comorbid illnesses noted in the US incident dialysis population over the past decade, it would be increasingly important to identify not only the presence, but also the severity of comorbid illnesses in quantifying individuals risk for adverse outcomes. The objective of the present study was to compare four commonly used comorbidity instruments, the ICED, the CCI, the Wright-Khan Index and the Davies Index, for their accuracy in predicting 1 year mortality in a large representative haemodialysis population.
![]() |
Subjects and methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Data collection
The Medical Information System (MIS) is an electronic medical record database used by all dialysis units throughout the DCI network. Nurses and other staff at the local dialysis units routinely enter clinical information relating to the care of the patient into the MIS, including progress notes, treatment sessions, medication lists, hospitalizations and reasons for admission. Monthly laboratory results are linked directly to the MIS. A quality management committee routinely monitors data for accuracy and completeness. Deaths are recorded by the dialysis unit staff and validated against data submitted to the Centers for Medicare and Medicaid Services (CMS) on the Death Notification Form (Form 2748). Transfer out of the dialysis unit, loss-to-follow-up and modality switches are also routinely recorded within the MIS. Prospective collection of comorbidity data ended February 1, 2001, to enable assessment of the feasibility of the programme and the validity of the risk assessment. The end of observation for these analyses was June 31, 2001.
Assessment of comorbidity using the ICED
Comorbidity information was abstracted and scored according to the definitions of the ICED. The ICED consists of 166 items that characterize both the presence and severity of 19 medical conditions and 11 physical impairments. Details of the instrument and its scoring are provided elsewhere [15]. In brief, it consists of two assessments: the Index of Disease Severity (IDS), based on data abstracted from the medical records, and an assessment of physical impairments [Index of Physical Impairments (IPI)], based on the dialysis nurses observation of the patient in the outpatient dialysis unit. To score the IDS, nurses were instructed to update and review discharge summaries, problem lists (routinely maintained in DCI patients), consultation letters, nurse and physician progress notes, medication lists, diagnostic imaging reports, and the nephrologists history and physical exam(s). The final ICED score is based on an algorithm, which combines the single peak disease category (IDS) with the single peak physical impairment category (IPI), ranging from 03 with 3 as the most severe. ICED 0 was combined with ICED 1 because the former was infrequent (<0.5%).
The CCI, Wright-Khan and Davies indices were scored retrospectively from the comorbidity database assembled above. The ICED includes all items within the other instruments for all conditions except malignancy, the latter is defined in more detail in the CCI. Thus, for these analyses we excluded subjects with a malignancy within the past 5 years (7% of the population). Item definitions and weights used were those defined for the original instruments [3,5,8,9]. For the CCI, the weighted items were summed and divided into levels as defined by Fried et al. [11]. We were able to define an additional higher severity level than the prior study due to our larger sample size and/or the greater comorbidity severity of our population. The time taken to review the medical record and complete the comorbidity assessment using each instrument was not recorded in this study; however, other studies have reported an average of 50 min per patient with the ICED [15,16] and 20 min with the CCI [11]. Completion time for the Wright-Khan and Davies indices has not been reported in prior studies, but is likely to be similar or less than for the CCI.
Statistical methods
The outcome of interest was death within 1 year of the ICED assessment. The start of the observation time was the date of the ICED assessment. Patients were censored at transplant, transfer to a non-participating clinic, or June 1, 2001, whichever came first. No patients were reported to recover kidney function. Eight patients were reported to have withdrawn from dialysis, although a death date was not reported despite more than 3 months follow-up after the censoring date in each case. These events may have been a recovery of renal function or a transfer from the unit. As a death was not recorded, the date of last observation was used for these eight subjects and a death was not counted. Baseline characteristics were described using means and SDs for continuous variables and frequencies for categorical variables. Differences across ICED strata (01, 2, 3) were tested for significance using one-way ANOVA for continuous variables and chi-squared tests for categorical variables.
Univariate associations of each instrument with 1 year mortality were assessed in separate logistic regression models with 1 year mortality as the binary outcome. Logistic regression was used as it requires fewer assumptions than Cox proportional hazards regression and the output of the model is readily translated into a 1 year probability of death. The CCI and Wright-Khan Index incorporate age in the scoring, but both instruments were noted to perform better when age was added as a separate covariate in a previous study [13], thus, age was added to models in the present study. Adjusted multivariable models were constructed through the sequential addition of vintage (defined as <1 year, 13 years and >3 years since the start of dialysis), race (white, black and other), the cause of ESRD (glomerulonephritis, hypertension, diabetes, other, polycystic kidney disease), and serum albumin. Serum albumin was averaged over 30 days prior and 60 days subsequent to the date of comorbidity data collection. The change in the fit of the model upon addition of each covariate was expressed as the change in the 2 log-likelihood test statistic over the prior model. The overall fit of the model is expressed as the change in the 2 log-likelihood test statistic over the null model. The interaction of incident status (4 months vs >4 months since start of dialysis) was tested with each comorbidity instrument to determine whether instruments performed differently in incident vs prevalent populations.
Assessing model discrimination and calibration
One year mortality was predicted from univariate and multivariable-adjusted models derived using each instrument. Predictive performance of each instrument was quantified as the area under the receiver-operating characteristic curves (AUC) [17,18]. The receiver-operating characteristic curve (ROC) plots the false-positive rate (x-axis) as the cut-off for a true-positive result (y-axis) is varied. An AUC of 1.0 indicates that the instrument correctly orders all possible pairs of patients, given one who dies and one who survives, with a higher predicted risk of death to the patient who dies, while an AUC of 0.50 suggests the instrument is no better than chance alone. Ninety-five percent confidence intervals (CIs) were estimated for AUCs with the assumption of non-parametric distribution [19]. To exclude the presence of bias at the lower or upper end of the risk spectrum, the proportions of observed vs expected deaths were compared across deciles of risk using the Hosmer-Lememshow chi-squared test [20]. Data were analysed using SPSS Version 10.0 (Chicago, IL).
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Baseline characteristics stratified by comorbid illness severity
The study population used in the analyses was representative of the US dialysis population: the combined mean age was 62 years, 47% were female, 30% were AfricanAmerican, and 44% had diabetes as the cause of ESRD. Laboratory values averaged over a 90 day window of the comorbidity assessment were as follows: the mean (SD) serum albumin was 3.70 (0.44) g/dl, the mean (SD) haematocrit was 34.5% (3.6) and 83% had a Kt/V 1.2. Approximately 40% of the patients had comorbidity assessments performed within 1 year of starting dialysis. With increasing comorbidity severity, as measured by the ICED, patients were older, a greater proportion were Caucasian, and had diabetes as the cause of ESRD (Table 1). Increasing comorbidity severity was also significantly correlated with decreasing serum albumin (r = 0.21, P<0.0001).
|
|
|
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The ICED, which had greater discriminatory ability than the other instruments considered on their own, provides direction for improving predictive accuracy for this outcome. The key difference of the ICED from the other instruments is the detail by which disease severity is classified, as illustrated with the example of congestive heart failure in Table 4. Given the high prevalence of many comorbid conditions [16] it is important to characterize severity, and not merely the presence of disease in identifying those at highest risk. Another unique component of the ICED is the assessment of physical limitations, which has repeatedly been shown to be a strong prognostic factor in dialysis patients [21].
|
The reliability, or consistency in scoring across reviewers, is as important as instrument validity, given that these instruments will be used in multicentre settings. The ICED is the only instrument of the four that has been tested and found to have good reliability when used in a multicentre setting [15]. The Wright-Khan and Davies indices have been used for the most part in single centre studies, and were scored by physicians. Agreement may be very different with more reviewers, as in a multicentre study, or with reviewers of different training or background, for example, nurses compared with physicians.
Despite the advantages of a detailed and comprehensive comorbidity review, we acknowledge that the 160 item ICED may be too burdensome to be practical for everyday use in busy dialysis units. We believe comorbid conditions of key prognostic significance must be identified, defined using specific criteria and weighted according to relationships with clinical outcomes in a contemporary dialysis population, to improve upon results seen in this study.
Finally, this study has focused on the validity of comorbidity instruments for predicting 1 year mortality, yet other clinical outcomes are important endpoints of clinical trials or cost-effectiveness analyses. For example, patients underlying comorbid illness burden also impacts on quality of life, independence in daily living and hospitalization use. Different conditions and weights may be needed to assess relationships of individual comorbidity items with these other outcomes, which deserve to be studied.
These analyses have some limitations. The agreement between a nurse and a physician reviewer was not assessed in this study, although nurses were provided with a training manual and a resource person (D.C.M., A.A.M.) was available to field questions related to comorbidity scoring throughout the course of the study. This is similar to past studies where good inter-rater reliability in scoring the ICED was found [15,16]. The exclusion of patients with malignancy (7% of the population) might reduce predictive performance of the CCI, relative to the ICED, as it was the single comorbid condition that was more detailed in its definition than the ICED. The exclusion of this small proportion of patients, most of whom had the lowest severity weighting for this category, would be unlikely to alter performance significantly. In addition, the definitions of the ICED were used to score the CCI, Wright-Khan and Davies indices. Because the ICED has at least as much detail as these other instruments, the scoring of other instruments from this database is likely to be more accurate than the results of a prior study [13]. We cannot, however, exclude the possibility that comorbidity scoring may have differed had the respective instruments been used to abstract data from the medical record. The contribution of novel inflammatory markers such as C-reactive protein or Il-6, that have been shown to be prognostic for outcomes, were not collected and could not be compared for their contribution to the models.
In comparing comorbidity instruments used in dialysis patients over the last decade, we find low predictive accuracy for 1 year mortality. Of the instruments assessed in this study, the ICED was more accurate than the others, but this advantage was no longer present after addition of routinely available factors including, age, the cause of ESRD, race, vintage and serum albumin. None of these instruments, either alone or in combination with other factors were sufficiently accurate to be used solely in clinical decision-making. The combination of comorbidity and other factors that define case-mix severity must be accounted for in comparing outcomes in clinical trials and quality assessment programmes. Further research to identify, define and weight the key comorbidity variables of prognostic significance in a dialysis population is needed to improve the accuracy of the risk assessment.
![]() |
Acknowledgments |
---|
Conflict of interest statement. None declared.
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|