1 Centre for Anaesthesia and Cognitive Function, Department of Anaesthesia, St Vincents Hospital, Melbourne, Australia. 2 School of Psychological Science, La Trobe University, Melbourne, Australia. 3 Department of Anaesthesia and Pain Management, Alfred Hospital, Melbourne, Australia
*Corresponding author: Department of Anaesthesia, St Vincents Hospital, Victoria Parade, Melbourne, Victoria 3065, Australia. E-mail: silberbs{at}svhm.org.au
Accepted for publication: February 1, 2004
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods. Fifty patients undergoing CABG surgery completed both a conventional and a computerized battery of tests before and 6 days after CABG surgery. Fifty age- and education-matched controls completed the same test batteries 6 days apart. The reliability and the sensitivity to postoperative cognitive decline were computed for each battery.
Results. Both test batteries detected postoperative cognitive decline 6 days after CABG surgery. For the computerized battery, the reliability of the reaction times (intraclass correlation 0.890.92) was greater than for any test from the conventional battery (intraclass correlation 0.560.71), although accuracy measures were less reliable (intraclass correlation 0.610.89). The computerized battery detected all the cases of POCD identified by the conventional test battery and also five cases that were classified as normal by the conventional tests.
Conclusion. Computerized tests are suitable for measuring cognitive change after CABG surgery and may detect change in a greater proportion of patients 6 days after CABG surgery than conventional neuropsychological tests.
Br J Anaesth 2004; 92: 81420
Keywords: complications, cognitive deficit; monitoring, neuropsychological testing; surgery, cardiovascular
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
A variety of neuropsychological tests assessing different cognitive functions have been used to define the disorder.5 6 To introduce uniformity into the assessment and definition of postoperative cognitive decline, specific sets of conventional neuropsychological tests have been recommended.7 The inclusion of tests such as the Rey Auditory Verbal Learning Test (RAVLT), the trail making tests (parts A and B) and the grooved pegboard test in these recommendations is based on findings that they consistently show sensitivity to postoperative cognitive decline (i.e. they must be able to detect change). The ability of neuropsychological tests to detect postoperative cognitive decline depends heavily on their sensitivity to detect change in cognitive function over time. The properties of the tests recommended for the detection of postoperative cognitive decline reveal common features that also make them useful for repeated administration and therefore for detection of change. The recommended tests all have relatively high testretest reliability, which provides an estimate of the stability of the test over time. They have short administration times, parallel versions, small or non-existent practice effects, they assess simple cognitive functions and are relatively portable. Each test also yields data that have a broad range of possible scores, rarely suffer from ceiling or floor effects and are at least of an ordinal level of measurement.5 6 8
It is appealing intuitively to propose that the computerization of cognitive tests can increase their reliability through the standardization of test administration and recording of responses. However, computerization per se is not sufficient for increased sensitivity to change. Rather, sensitivity to change occurs if the features of the tests (computerized or conventional) required for detecting change can be enhanced. In addition to standardization of administration, computerization allows the rapid generation of infinite numbers of stimulus sets, and allows the accurate measurement of reaction time and accuracy while keeping the time for administration to a minimum.912 As with conventional tests, practice effects can be minimized if the actual tests are kept simple.9 A test battery designed explicitly to detect change has been shown to be more sensitive to cognitive change than conventional neuropsychological tests after sports injury, concussion and fatigue.1214 The salient features include minimal practice effect, suitability for repeated administration (e.g. multiple alternative forms, standardized stimulus sets and response requirements and short administration times) and the generation of normally distributed performance scores.
The aim of the present study was to determine whether this computerized test battery could detect postoperative cognitive decline after CABG surgery in the immediate postoperative period and to compare the sensitivity with that of conventional neuropsychological tests.
![]() |
Methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The control group consisted of 50 healthy people with normal medical and psychiatric function, selected to match the CABG group for age, years of education and baseline IQ (Table 1). The controls were a subset of a larger group (n=254) recruited through advertisements for volunteers to study cognitive changes associated with normal ageing. All were deemed to have normal cognitive function after completing the Consortium to Establish a Registry for Alzheimers Disease (CERAD) neuropsychological battery.15
|
Both the control and CABG groups completed a battery of conventional tests followed by the computerized tests. In the CABG group, neuropsychological assessment was undertaken before surgery (pre-admission clinic or day before) and before discharge (median 6 days, range 57 days). The control group was tested on the two test batteries approximately 6 days apart (median 6 days, range 59 days) to correspond with the CABG group.
The conventional tests were selected because they had been commonly used and recommended in the consensus statements.7 This battery (see Appendix in Supplementary data online) consisted of the CERAD Word Learning test, Symbol Digit Modalities test, Trail Making test parts A and B, Semantic Fluency test, and the Grooved Pegboard test.11 The conventional tests were scored according to standard protocols. Baseline intelligence (IQ) was estimated with the National Adult Reading Test.16
The computerized test battery (see Appendix in Supplementary data online), a subset of tests from the CogStateTM (Melbourne, Australia) battery,17 was selected to be brief and deliver the same number of performance measures as the conventional tests. For each test, the stimuli consisted of playing cards. Six test results were derived from the reaction time and accuracy of the three tests administered (detection, identification and matching). For each of the three tasks, 15 trials were presented.
For each computerized test, the number of trials on which an incorrect answer was given and the remaining correct trials were calculated and expressed as a percentage of the total trials. Trials on which incorrect responses occurred were then excluded from further analysis. Reaction times <100 ms and >1000 ms were classified as abnormally fast or slow responses and excluded.9 17 The reaction time distributions for each test showed significant negative skew (i.e. skew/SE skew >1.96); therefore, these were normalized using a logarithmic base 10 transformation. The mean of the log10 reaction time was calculated for each participant for each task.
The testretest reliability of all tests was calculated from the control sample tested at baseline and at day 6 using the intraclass correlation. The intraclass correlation is interpreted in the same way as Pearsons product moment correlation; however, unlike Pearsons r, it is robust to order effects (e.g. the practice effect).18 The testretest reliability is an estimate of the amount of error of measurement that is contained in a specific test. Therefore, testretest reliability in controls provides an estimate of the extent to which a test provides a stable measure of performance over time. This is important for tests used to measure change, because the more performance can vary by chance, the harder it is to detect true change.
For both test batteries, the presence of postoperative cognitive decline for each test was determined at the group level19 by submitting each participants score on each test to a series of 2 (group: control, CABG) x 2 (time: baseline, day 6) repeated measures analyses of variance. For all of the analyses conducted at a group level, statistical significance was set at P<0.01.
The hypothesis that cognitive change would be greater in the CABG group than in controls was tested by the presence of a significant group x time interaction. Significant interactions were investigated with matched pairs t-tests. The magnitude of change detected in both the control and CABG group on all tests was computed using Cohens d statistic.20 This statistic expresses the differences between means as a function of their pooled standard deviation. It therefore provides a common metric for evaluating the magnitude of change on tests with different units and can be compared with published conventions that specify whether an effect is small (0.2), medium (0.5) or large (0.8).
In addition to group change, the incidence of cognitive impairment in individuals for both test batteries was determined by calculating the reliable change index (RCI).6 21 To calculate the RCI, the difference in performance from the baseline to the day-6 assessment was computed for each test in each of the control subjects. The mean and standard deviation (SD) of the distributions of difference scores for each test were then established. The mean of the difference scores from the control group reflects the expected effect of practice for each test, while the SD of the difference scores provides an estimate of the expected variability in normal performance on the cognitive tests over the study period. For each subject, assessment at the baseline and day 6 for each test was used to calculate the RCI as follows: RCI=(day 6 scorebaseline score)(practice effect estimated from controls)/(SD of difference scores estimated from control group).
An RCI can be interpreted like a standard (Z) score. For each test, a deterioration in RCI 1.65 was classified as abnormal decline in cognitive performance. The cut-off of RCI=1.65 is the point beyond which 5% of the values from the normal sample population will fall (i.e. P<0.05, one-tailed). To further protect against Type I error, postoperative cognitive decline was defined only if an individual showed cognitive decline on 2 or more performance measures derived from the tests in each battery.
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
The testretest reliability for conventional and computerized tests in the control group is shown by the intraclass correlations in Table 3. The computerized tests were more reliable than the conventional tests. Within the computerized battery, measures of the reaction time were more reliable than measures of accuracy. For the conventional tests the grooved pegboard had the highest reliability.
|
|
|
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The properties of the computerized test battery satisfy most of the requirements for detecting cognitive change. Indeed, the computerized battery has been designed specifically for detecting change in contrast to the conventional tests, many of which were originally designed for single administration. Thus, the computerized tests are sensitive to change, avoid practice effects, and are not prone to floor and ceiling effects. In addition, the computerized battery has other suitable characteristics for testing after surgery. It may overcome cultural and language difficulties and is easy to administer, standardized and quick. The results can be downloaded and analysed automatically, eliminating human error in both assessing and transcribing. In view of both the neuropsychological properties and utility of the computerized battery, it seems opportune to compare its use with that of the standard conventional testing.
The present results are consistent with previous studies showing cognitive decline in the immediate postoperative period after CABG surgery.19 24 Both the conventional and computer test batteries detected postoperative cognitive decline 6 days after CABG surgery.
Analysis of the intraclass correlations indicated that, for the computerized battery, the reliability of all three reaction times was higher than for any test from the conventional battery. There were minimal practice effects for reaction time or accuracy on the computer measures. The accuracy measures were less reliable than the reaction times, although the reliability of identification and matching accuracy were equivalent to those of the Trail Making and Symbol Digit Modalities tests. The low reliability of the accuracy measure from the identification task occurred because participants in both groups gave performance scores close to the maximum (100%) at both the preoperative and postoperative assessments (Table 3).
At the group level, the computerized battery showed cognitive decline for detection, identification and matching, and matching accuracy. The only conventional test to show a significant effect of CABG surgery was the grooved pegboard test. This pattern of results is related to the testretest reliability, as only measures with reliabilities >0.7 gave results that indicated a significant effect of the surgery at the group level. The importance of choosing an appropriate test battery is best demonstrated by referring to a previous study by Mullges et al.19 that used conventional tests to detect postoperative cognitive decline. Deterioration in cognitive performance at a group level was noted 3 days after CABG surgery but performance returned to the baseline level by day 6. This was interpreted as showing that postoperative cognitive decline resolved relatively quickly after CABG surgery.1 19 However, our results suggest that the Trail Making A and B tests may not be sensitive enough to detect the cognitive decline that is present 6 days after CABG surgery. In the present study no decline was detected on the Trail Making A and B Tests 6 days after CABG surgery. However, a significant decline in performance was observed for the reaction time tests from the computerized battery and the grooved pegboard test.
The ability of any test to detect true change in cognitive functions depends on its ability to resist practice effects and the extent to which the variability within individuals, over time, is kept to a minimum.6 9 The use of a healthy control group to derive estimates of a practice effect, variability in change scores and the reliability of the tests is consistent with other studies of postoperative cognitive decline after CABG surgery21 and anaesthesia.8 RCI in which the denominators are estimated from the analysis of change in a non-operated control group provide the most reliable and valid method for determining the significance of any cognitive change within individuals.6 Minimal practice effects were found for the computerized tests in the control group. This is consistent with a previous study in which no practice effects were observed when the same tests were given four times to a group of healthy older people.9 Importantly, the tests in the CogState battery were designed explicitly to minimize practice effects by restricting the assessment to very simple attention, memory and executive functions.
In this study, the conventional battery identified postoperative cognitive decline in 32% of participants at day 6. The rate of postoperative cognitive decline was 42% of subjects when calculated from performance on the computerized test battery. Both estimates of postoperative cognitive decline are consistent with previously reported studies in CABG surgery populations.1 3 5 If the conventional tests were considered as the gold standard, then the sensitivity of the computer tests would be 100% and the specificity 85%. However, as noted by Arrowsmith et al. because of the difficulties in psychometric testing,25 there is no gold standard against which to compare new tests and the comparison against the conventional tests must be interpreted in this light.
The computerized cognitive test battery represents a viable alternative to conventional testing. All of the cases of postoperative cognitive decline classified by the conventional battery were also classified by performance on the computerized battery, although five additional cases were identified by their performance on the computerized battery. The question arises as to whether these five additional cases of postoperative cognitive decline were true cases or false positives. We believe that the additional cases of postoperative cognitive decline identified on the computerized battery may be true cases. This is because the greater sensitivity of the computerized tests to postoperative cognitive decline is consistent with their high testretest reliability, absence of practice effects and relatively small variability found in the difference scores generated in the control group. Secondly, the same computerized tests have been shown to possess greater sensitivity to subtle cognitive decline than conventional tests in other circumstances.13 14 17 Finally, the five additional cases of postoperative cognitive decline identified by the computerized test battery all showed some impairment on conventional testing (three patients deteriorated >1.65 SD on one conventional test and >1.0 SD on at least one other; two deteriorated >1.1 SD on two or more of the tests).
The concept of specific neuropsychological domains being affected and impaired during surgery is appealing.26 However, most tests require multiple cognitive processes for successful performance. Therefore, the disruption of different cognitive systems may present as the same cognitive deficit.5 Thus, the current measurement tools reliably and sensitively measure change and make no inference as to which neuronal pathways or resultant cognitive domains may be disturbed.
In conclusion, we have compared the use of a computerized test battery with conventional tests to detect postoperative cognitive decline after CABG surgery. The computerized tests provided suitable test properties, were easy to administer and analyse, and proved to be sensitive to postoperative cognitive decline after CABG surgery. The use of the computerized battery warrants further investigation in the detection of postoperative cognitive decline after CABG surgery and has the potential to overcome many of the methodological difficulties that have hampered research in this area.
![]() |
Supplementary data |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
Acknowledgements |
---|
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
2 Murkin JM. Perioperative neuropsychologic testing. J Cardiothorac Vasc Anesth 2001; 15: 13[CrossRef][ISI][Medline]
3 Selnes OA, Goldsborough MM. Cognitive changes after coronary artery bypass surgery. Curr Opin Psychiatry 2002; 15: 28590[CrossRef][ISI]
4 Newman MF, Kirchner JL, Phillips-Bute B, et al. Longitudinal assessment of neurocognitive function after coronary-artery bypass surgery. N Engl J Med 2001; 344: 395402
5 Symes E, Maruff P, Ajani A, Currie J. Issues associated with the identification of cognitive change following coronary artery bypass grafting. Aust N Z J Psychiatry 2000; 34: 77084[CrossRef][ISI][Medline]
6 Collie ADD, Falleti MG, Silbert BS, Maruff P. Determining the extent of cognitive change after coronary surgery: a review of statistical procedures. Ann Thorac Surg 2002; 73: 200511
7 Murkin JM, Newman SP, Stump DA, Blumenthal JA. Statement of consensus on assessment of neurobehavioral outcomes after cardiac surgery. Ann Thorac Surg 1995; 59: 128995
8 Rasmussen LS, Larsen K, Houx P, Skovgaard LT, Hanning CD, Moller JT. The assessment of postoperative cognitive function. Acta Anaesthesiol Scand 2001; 45: 27589[CrossRef][ISI][Medline]
9 Collie A, Maruff P, Darby DG, McStephen M. The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief testretest intervals. J Int Neuropsychol Soc 2003; 9: 41928[CrossRef][ISI][Medline]
10 Lowe C, Rabbitt P. Test/re-test reliability of the CANTAB and ISPOCD neuropsychological batteries: theoretical and practical issues. Cambridge Neuropsychological Test Automated Battery. International Study of Post-Operative Cognitive Dysfunction. Neuropsychologia 1998; 36: 91523[CrossRef][ISI][Medline]
11 Lezak M. Neuropsychological Assessment. Oxford: Oxford University Press, 1995
12 Collie A, Darby D, Maruff P. Computerised cognitive assessment of athletes with sports related head injury. Br J Sports Med 2001; 35: 297302
13 Makdissi M, Collie A, Maruff P, et al. Computerised cognitive assessment of concussed Australian Rules footballers. Br J Sports Med 2001; 35: 35460
14 Maruff PFM, Collie A, Darby D, McStephen M. Fatigue related impairment in the speed, accuracy, and variability of psychomotor performance: comparison with blood alcohol levels. J Sleep Res. In press
15 Collie A, Maruff P, Currie J. Behavioral characterization of mild cognitive impairment. J Clin Exp Neuropsychol 2002; 24: 72033[CrossRef][ISI][Medline]
16 Nelson H. The National Adult Reading Test (NART): Test Manual. Windsor, UK: NFER-Nelson, 1982
17 Darby D, Maruff P, Collie A, McStephen M. Mild cognitive impairment can be detected by multiple assessments in a single day. Neurology 2002; 59: 10426
18 Bland JM, Altman DG. Measurement error and correlation coefficients. Br Med J 1996; 313: 412
19 Mullges W, Berg D, Schmidtke A, Weinacker B, Toyka KV. Early natural course of transient encephalopathy after coronary artery bypass grafting. Crit Care Med 2000; 28: 180811[CrossRef][ISI][Medline]
20 Zakzanis KK. Statistics to tell the truth, the whole truth, and nothing but the truth: Formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers. Arch Clin Neuropsychol 2001; 16: 65367[CrossRef][ISI]
21 Kneebone AC, Andrew MJ, Baker RA, Knight JL. Neuropsychologic changes after coronary artery bypass grafting: use of reliable change indices. Ann Thorac Surg 1998; 65: 13205
22 Ancelin MRG, Ledesert B, Bonnel F, Cheminal J, Ritchie K. Exposure to anaesthetic agents, cognitive functioning and depressive symptomatology in the elderly. Br J Psychol 2001; 178: 3606
23 Fearn SJ, Pole R, Wesnes K, Faragher EB, Hooper TL, McCollum CN. Cerebral injury during cardiopulmonary bypass: emboli impair memory. J Thorac Cardiovasc Surg 2001; 121: 115060
24 Silbert BS, Scott DA, Doyle TJ, et al. Neuropsychologic testing within 18 h after cardiac surgery. J Cardiothorac Vasc Anesth 2001; 15: 204[CrossRef][ISI][Medline]
25 Arrowsmith JE, Grocott HP, Reves JG, Newman MF. Central nervous system complications of cardiac surgery. Br J Anaesth 2000; 84: 37893[Abstract]
26 McKhann GM, Goldsborough MA, Borowicz LM, et al. Cognitive outcome after coronary artery bypass: a one-year prospective study. Ann Thorac Surg 1997; 63: 5105