Validation of a clinical prediction rule to reduce preoperative type and screen procedures{dagger} {dagger}{dagger}

W. A. van Klei*,1,2, K. G. M. Moons1,2, A. T. Rheineck-Leyssius3, C. J. Kalkman1, C. L. G. Rutten4, J. T. A. Knape1 and D. E. Grobbee2

1 Department of Peri-operative Care, Anaesthesia and Pain Management, 2 Julius Centre for General Practice and Patient Oriented Research, University Medical Centre Utrecht, PO Box 85500, 3508 GA Utrecht, The Netherlands. 3 Twenteborg Hospital, Department of Anaesthesiology and Intensive Care Medicine, Zilvermeeuw 1, 7609 PP Almelo, The Netherlands. 4 Isala Clinics, Department of Anaesthesiology, Weezenlanden Hospital, PO Box 10500, 8000 GM Zwolle, The Netherlands*Corresponding author

{dagger}Presented in abstract form at the ASA annual meeting, October 2001, New Orleans, USA.
{dagger}{dagger}This article is accompanied by Editorial II.

Accepted for publication: January 16, 2002


    Abstract
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Background. We have developed a prediction rule for the occurrence of perioperative red blood cell transfusion to help to reduce the number of unnecessary preoperative type and screen procedures. We evaluated the robustness of this prediction rule in patients from another hospital.

Methods. The rule was retrospectively applied to 1282 consecutive patients (‘validation set’) who underwent similar surgical procedures to the patients in the derivation study. The outcome was similarly defined as any allogeneic transfusion on the day of surgery or during the first postoperative day. The predictive value of the rule was assessed using a Receiver Operating Characteristic curve (ROC) and compared with the results of the derivation study. Subsequently, the number of correctly predicted transfusions was compared.

Results. The patient characteristics did not differ between the two sets, except for the incidence of transfusion (derivation study: 18%; present study: 8%). In the validation set, the ROC area of the prediction rule was 0.78 (95% confidence intervals [CI]: 0.73–0.82), which was within the CI of the ROC area found in the derivation study (0.75; 95% CI: 0.72–0.79). In total, 35% of the type and screen procedures could be omitted (derivation study: 50%), with 13% missed transfused patients (derivation study: 20%).

Conclusions. After comparing the results of this validation study with that of the derivation study, the prediction rule was robust and may work in other clinics as well.

Br J Anaesth 2002; 88: 221–5

Keywords: blood, transfusion; prediction, preoperative


    Introduction
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Each surgical patient is assessed by an anaesthesiologist before surgery. This preoperative evaluation usually consists of a medical history, physical examination and, if necessary, additional tests.1 However, money can be wasted on inappropriate additional tests.1–7 For example, most patients who are typed and screened before surgery do not require a transfusion.

We have developed a clinical prediction rule based on simple patient characteristics to predict blood transfusion in patients undergoing surgery with an intermediate risk for transfusion (1–30%).8 With this rule, the number of type and screen procedures performed before surgery could be reduced by about 50%, with an acceptable number of missed transfused patients.

We wished to determine whether the rule could be adopted by other clinics. In this validation study we aimed to evaluate the robustness of our prediction rule in patients from another hospital, which should be done before implementing a prediction rule in clinical practice.9–11


    Methods
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Prediction rule
In a previous study at a non-university hospital (the derivation study) we included 1482 patients who underwent surgery having intermediate transfusion risk (1–30%).8 We developed a prediction rule for the occurrence of perioperative red blood cell (RBC) transfusion. The rule aimed to reduce the number of unnecessary type and screen procedures performed before surgery. Table 1 shows the contents of the rule, that is: 1*gender+1*age>=70+(1, 2, 4 or 5)*surgery procedure. For each patient a score can be estimated in which female sex and age >=70 yr count for 1 point and scheduled surgery procedure for 1, 2, 4 or 5 points, depending on the procedure. The surgical procedures were allocated into five categories (see the legend for Table 1).8 A threshold value of 2 was introduced, in which <=2 indicated that ‘transfusion will not occur, and a preoperative type and screen procedure can be withheld.’ Using this threshold, in 35% of the patients a type and screen could be omitted, with 16% missed transfused patients. Subsequently, in the subgroup of patients with score >2 a preoperative haemoglobin concentration (preopHb) at a threshold of 14 g dl–1 was used to further reduce the number of type and screen procedures. A preopHb >=14 g dl–1 indicated ‘transfusion will not occur; do not type and screen’ and <14 g dl–1 indicated ‘type and screen’. Doing so, the number of type and screen procedures could be reduced by about 50%, with 20% missed transfused patients in total.8


View this table:
[in this window]
[in a new window]
 
Table 1. Components of the rule, with corresponding scores and original regression coefficients (ß).8 The intercept (constant) was –3.70 (95% confidence intervals [CI]: –4.67; –2.73). aThe score of each predictor was obtained by dividing the corresponding regression coefficient by the smallest coefficient (0.52) and rounded to the nearest integer. bGroup 1: laparoscopic cholecystectomy; Group 2: mastectomy and transurethral resection of tumour (TURT) or prostate (TURP); Group 3: open cholecystectomy, vaginal hysterectomy, Caesarean section, surgery for urinary incontinence and vaginal prolapse; Group 4: non-cardiac thoracic surgery (e.g. lobectomy), vascular (arterial) surgery (e.g. femoropopliteal bypass), prostate enucleation and endometrial cancer surgery; Group 5: abdominal and supravaginal hysterectomy, hip fracture surgery, revision knee prosthesis, leg amputation, gastro-enterostomy, colon resection and radical abdominal hysterectomy. ß, regression coefficient of the logistic model. Group 1 is the reference group
 
Patients
To determine the accuracy of the rule, it was retrospectively applied to 1282 consecutive patients (aged 18–103 yr). These patients underwent the relevant surgical procedures (see the legend to Table 1) in 1998 at the University Medical Centre Utrecht, a 1080-bed teaching hospital in The Netherlands (the ‘validation set’). All patients were typed and screened before surgery as the routine practice.

Outcome
The outcome in the present study was as in the derivation study: the need for any allogeneic RBC transfusion, defined as transfusion of one or more units of packed cells on the day of surgery or on the first day after surgery. The transfusion decision was made by individual clinicians (anaesthesiologists and surgeons), who were unaware of the prediction rule value, the rule being validated retrospectively. In general, blood was given when the haemoglobin concentration was less than 8 g dl–1.

Data collection
After approval of the hospital Ethics Committee, data were collected from the hospital information system. There were no missing data on any of the predictor or outcome variables, except that preopHb had not been determined in 245 patients (19%). Surgical procedures were allocated to five subgroups, as in the derivation study.

Analysis
SPSS Release 10.1 for Windows was used for the analysis. The discriminative value of the prediction rule (Table 1) was assessed using the area under the Receiver Operating Characteristic curve (ROC area) and compared with the ROC area of the rule in the derivation study.8 12 Subsequently, the same threshold value as used in the derivation study (<=2 points) was used to compare the number of correctly predicted transfused and not transfused patients with those in the derivation study. Finally, the same threshold preopHb value was used (14 g dl–1) in all patients with score >2, and the number of correctly predicted and missed transfusions was compared.


    Results
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
There were no major differences in the patient characteristics of the derivation and validation studies, except for the incidence of transfusion, which was 18% in the derivation study and 8% in the validation set (Table 2). In the validation set the ROC area of the prediction rule was 0.78 (95% confidence interval [CI]: 0.73–0.82) (Fig. 1). This area was within the 95% CI of the ROC area found in the derivation study (0.75; 95% CI: 0.72–0.79).


View this table:
[in this window]
[in a new window]
 
Table 2. Patient characteristics of derivation8 and validation sets. Values are numbers (percentage) except amean age (SD); bsurgical procedures are listed in Table 1
 


View larger version (18K):
[in this window]
[in a new window]
 
Fig 1 ROC curves of the transformed prediction rule (Table 1). Each point indicates a score threshold from 0 (upper right) to 7 (bottom left).

 
Table 3 shows the number of transfused and non-transfused patients across score categories of the rule. Applying the score threshold of >2, type and screen tests would be omitted in 23% of the patients, with 8% missed transfused patients (derivation study: 35% and 16%, respectively). Consequently, using the threshold of >2 the specificity was 24% (283/1182) and the sensitivity 92% (60+32/100), compared with 40% and 84%, respectively, in the derivation study. Reading the table horizontally, one can estimate the sensitivity and specificity of the rule for various thresholds. The sensitivity and specificity of all possible score thresholds can be obtained from the ROC curve (Fig. 1). Reading down Table 3 provides the predictive values per score. Of all 291 patients with score <=2, 283 patients were indeed not transfused, yielding a negative predictive value of 97% (derivation study: 90%). In the group of patients with score >2, 92 of the 991 patients were indeed transfused, a positive predictive value of 9% (derivation study: 27%).


View this table:
[in this window]
[in a new window]
 
Table 3. Distribution of transfused and non-transfused patients according to the score of the rule (and corresponding risk of transfusion). Values are absolute numbers (percentage of total). aCategories of the score as estimated from the clinical scoring rule. bRisk or probability of transfusion as estimated by the untransformed prediction rule given in the third column of Table 1: Risk=1/(1+exp –[–3.701+0.629*gender+0.546*age>=70+0.524*group 2+1.291*group 3+2.287*group 4+2.386*group 5]). n=number of subjects per score (risk) category
 
Table 4 shows the distribution of patients with score >2 across the two categories of preopHb. Of the 991 patients with score >2, 193 had missing values. These missing data were distributed equally among patients with (10%) and without (7%) transfusion (P=0.13, likelihood ratio test). Therefore, they were excluded from the analysis. A further reduction in type and screen investigations of 15% (derivation study: 24%) could be achieved by withholding type and screen procedures in all patients with a preopHb concentration >=14 g dl–1, at the expense of another five missed transfusions.


View this table:
[in this window]
[in a new window]
 
Table 4. Distribution of transfused and non-transfused patients according to the preoperative haemoglobin (Hb) concentration in the patients from Table 3 with score >2. Values are as absolute numbers (percentage of total). n=number of subjects per Hb category
 
In total, after applying the rule and the preopHb threshold to the validation set, 35% of the type and screen procedures could be omitted (derivation study: 50%), with 13 (13%) missed transfused patients (derivation study: 20%). These patients required an average of 2.7 units RBC per subject (95% CI: 2.0–3.4); six patients required more than two units (Table 5).


View this table:
[in this window]
[in a new window]
 
Table 5. Surgery and transfusion characteristics of transfused patients (n=13) with a score >2 and a preoperative haemoglobin concentration <=14 g dl–1 (‘missed transfused patients’). aUnits of red blood cells per patient. TUR, transurethral resection of prostate or tumour. n=number of patients
 

    Discussion
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
We tested our rule to predict perioperative RBC transfusions and reduce the number of type and screen procedures, in new patients undergoing identical surgery from another hospital. In total, 35% of the preoperative type and screen procedures could be omitted, at the expense of 13% missed transfused patients. These results are comparable with the numbers found in the derivation study.8

To appreciate these findings, it should first be noted that the rule applies only to patients scheduled for the surgical procedures included in the rule. Second, in this validation study the incidence of transfusion (8%) was substantially lower than in the derivation study (18%). This is probably a reflection of the haemoglobin concentration transfusion threshold of 8 g dl–1 used in the present study, compared with a threshold of 10 g dl–1 in the derivation study. The value of a prediction rule may be affected by differences in incidence.13–15 We estimated the performance of the rule after adjusting for the difference in transfusion incidence, that is, after adjusting the intercept of the original logistic regression model from which the scoring rule was derived (Table 1).8 However, this adjustment showed no effect on the ROC area and did not improve the predictive accuracy in terms of absolute numbers proportions (probabilities), as shown in Tables 3 and 4. We therefore believe that adjustment for differences is not necessary in the scoring rule. Third, 19% of the preopHb values were missing. The missing data were randomly distributed over the outcome. Hence, we think their exclusion has not biased the results shown in Table 4. Fourth, the acceptability of the 13% of transfusions that were not predicted (Table 5) must be considered. Possibly, patients who received two units or fewer could be typed and screened during the surgery, and colloids could be administered in the meantime. The same could have been done, in the six patients who required more than two units, and O-group blood could have been administered in an emergency. In our previous paper we discussed administering O-group blood, given the low prevalence of irregular antibodies in the general population (2.5%).8 Although one can argue against administering O-group blood in non-emergency operations, we estimated that irregular antibodies can be a problem in only 0.1% of all transfusions among surgical procedures with intermediate transfusion risk.8 Finally, the rule was derived and validated in a general hospital and, in the present study, validated in a university hospital. Since the test performed well in both types of hospital, we conclude that the prediction rule is robust and is likely to work in both settings.

Several prediction rules for perioperative blood transfusion have been developed already, mainly in orthopaedic surgery.16–20 As far as we know, only one study validated a scoring system for predicting blood transfusion as we have done.21 In that study, the accuracy of a scoring rule for predicting blood transfusion following hip or knee replacement (containing surgical procedure, preopHb and weight) was prospectively evaluated at two different clinics and judged as reasonable, with ROC areas of 0.78 and 0.79. These results are comparable with those found in our study, but our rule applies to a wider range of surgical procedures. Most prediction models for perioperative blood transfusion cover a small range of surgical procedures.16 However, it would be desirable to derive and validate a prediction model that covers all types of surgery (procedures with low, intermediate and high risk for transfusion) and to evaluate whether additional predictors play a role.

In conclusion, the previously derived rule to predict the need for blood transfusion in surgical procedures with intermediate transfusion risk can be applied in other clinics as well. As our rule aimed to reduce type and screen procedures before surgery, the use of the rule could reduce the costs of perioperative patient care. Assuming that the average direct cost of type and screen procedures is about US$80, the application of our rule will lead to a cost reduction of about US$3 million dollar per 100 000 surgical procedures with intermediate transfusion risk (35%x100 000x$80),8 although this reduction in cost will be somewhat lower when the cost of measuring haemoglobin concentration is taken into account.


    References
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
1 Roizen MF. Preoperative evaluation. In: Miller RD, ed. Anaesthesia. New York: Churchill Livingstone, 2000; 824–83

2 Schein OD, Katz J. The value of routine preoperative testing before cataract surgery. N Engl J Med 2000; 342: 168–75[Abstract/Free Full Text]

3 Haug RH, Reifeis RL. A prospective evaluation of the value of preoperative laboratory testing for office anesthesia and sedation. J Oral Maxillofac Surg 1999; 57: 16–20[ISI][Medline]

4 Narr BJ, Warner ME, Schroeder DR, Warner MA. Outcomes of patients with no laboratory assessment before anesthesia and a surgical procedure. Mayo Clin Proc 1997; 72: 505–9[ISI][Medline]

5 Pollard J, Zboray A, Mazze R. Economic benefits attributed to opening a preoperative evaluation clinic for outpatients. Anesth Analg 1996; 83: 407–10[ISI][Medline]

6 Perez A. Value of routine preoperative tests: a multicentre study in four general hospitals. Br J Anaesth 1995; 74: 250–6[Abstract/Free Full Text]

7 Narr BJ, Hansen T, Warner ME. Preoperative laboratory screening in healthy Mayo patients: Cost-effective elimination of tests and unchanged outcomes. Mayo Clin Proc 1991; 66: 155–9[ISI][Medline]

8 vanKlei WA, Moons KGM, van Rheineck Leyssius AT, Rutten CLG, Knape JTA, Grobbee DE. A reduction in Type and Screen: preoperative prediction of RBC transfusions in surgery procedures with intermediate transfusion risks. Br J Anaesth 2001; 87: 250–7[Abstract/Free Full Text]

9 Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med 1985; 313: 793–9[Abstract]

10 Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested methods of methodological standards. JAMA 1997; 277: 488–94[Abstract]

11 Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999; 130: 515–24[Abstract/Free Full Text]

12 Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic curve. Radiology 1982; 143: 29–36[Abstract]

13 Wigton RS, Connor JL, Centor RM. Transportability of a decision rule for the diagnosis of streptococcal pharyngitis. Arch Intern Med 1986; 146: 81–3[Abstract]

14 Poses RM, Cebul RD, Collins LM, Fages SS. The importance of disease prevalence in transporting clinical prediction rules. Ann Intern Med 1986; 105: 586–91[ISI][Medline]

15 Moons KGM, van Es G, Michel BC, Buller HR, Habbema JDF, Grobbee DE. Redundancy of Single Diagnostic Test Evaluation. Epidemiology 1999; 10: 276–81[ISI][Medline]

16 Grosflam JM, Wright EA, Cleary PD, Katz JN. Predictors of blood loss during total hip replacement surgery. Arthritis Care Res 1995; 8: 167–73[Medline]

17 Keating EM, Meding JB, Faris PM, Ritter MA. Predictors of transfusion risk in elective knee surgery. Clin Orthop 1998; 357: 50–9[Medline]

18 Larocque BJ, Gilbert K, Brien WF. A point score system for predicting the likelihood of blood transfusion after hip or knee arthroplasty. Transfusion 1997; 37: 463–7[ISI][Medline]

19 Magovern JA, Sakert T, Benckart DH, et al. A model for predicting transfusion after coronary artery bypass grafting. Ann Thorac Surg 1996; 61: 27–32[Abstract/Free Full Text]

20 Nuttall GA, Santrach PJ, Oliver-WC J, et al. The predictors of red cell transfusions in total hip arthroplasties. Transfusion 1996; 36: 144–9[ISI][Medline]

21 Larocque BJ, Gilbert K, Brien WF. Prospective validation of a point score system for predicting blood transfusion following hip or knee replacement. Transfusion 1998; 38: 932–7[ISI][Medline]