Institute of Psychiatry, London
Department of Psychiatry, Yale University, New Haven, Connecticut, USA
Institute of Psychiatry, London
Institute of Psychiatry/Maudsley Hospital, London
Queen Mary College, University of London
Institute of Psychiatry, London, UK
Correspondence: Professor Matthew Hotopf, Academic Department of Psychological Medicine, Institute of Psychiatry, Weston Education Centre, 10 Cutcombe Road, London SE5 9RJ, UK. Tel: +44(0)2078480778; fax: +44(0)2078485408; e-mail: m.hotopf{at}iop.kcl.ac.uk
Declaration of interest None. The study was funded by the WellcomeTrust.
See pp.
379385, this issue.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Aims To describe the interrater reliability of two independent assessments of capacity to consent to treatment, as well as assessments made by a panel of clinicians based on the same interview.
Method Fifty-five patients were interviewed by two interviewers 17 days apart and a binary (yes/no) capacity judgement was made, guided by the MacArthur Competence Assessment Tool for Treatment (MacCATT). Four senior clinicians used transcripts of the interviews to judge capacity.
Results There was excellent agreement between the two interviewers for capacity judgements made at separate interviews (kappa=0.82). A high level of agreement was seen between senior clinicians for capacityjudgements of the same interview (mean kappa=0.84).
Conclusions In combination with a clinical interview, the MacCATT can be used to produce highly reliable judgements of capacity.
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
It is against this background that attention has turned towards the assessment of mental capacity in individuals with mental disorder. The Expert Committee that advised the British Government on reform of the England and Wales Mental Health Act 1983 suggested that capacity should be a significant criterion in a new Mental Health Act (Expert Committee, 1999). This would bring mental health legislation more in line with established principles governing other healthcare decisions. In general, an individual would have to lack capacity before involuntary powers could be used and this absence of capacity would presumably have to be established on the basis of the independent judgements of two mental health clinicians applying the same test. Although the recommendation was rejected and was not included in the original or revised Draft Mental Health Bill (Department of Health, 2002, 2004), one criticism of a capacity-based Mental Health Act has been that assessments of capacity in the mental health setting are no less fraught than those of say, risk or treatability (Fulford & Sayce, 1998).
![]() |
METHOD |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
The local research ethics committee approved the study. After complete description of the study to the participants, written informed consent was obtained. There are potential problems in conducting research on patients who may lack the capacity to consent (Doyal, 1997; Osborn, 1999; Gunn et al, 2000). A small number of patients were deemed too disturbed to participate by medical or senior nursing staff and it was therefore not possible to infer assent to the study. Other reasons for exclusion were being on no prescribed psychotropic medication or receiving medication for the sole purpose of a medically assisted alcohol detoxification, and speaking no English.
Measurement of capacity
The MacCATT was administered to the patient in both interviews. It
is a semi-structured interview that provides relevant treatment information
for the patient and evaluates capacity in terms of its different components.
As such it can detect impairment in four areas: the patients
understanding of the disorder and treatment-related information;
appreciation of the significance of that information for the patient,
in particular the benefits and risks of treatment; the reasoning
ability of the patient to compare their prescribed treatment with an
alternative treatment (and the impact of these treatments on their everyday
life); and ability of the patient to express a choice between their
recommended treatment and an alternative treatment. The interview was modified
for the purpose of our study. Instead of offering an alternative treatment,
patients were given the option of no treatment as the
alternative to their prescribed or recommended medication. This
was to avoid confusion about the patients current treatment and also to
prevent potential problems in the relationship between the participant and the
treating clinician. This constituted another sub-scale understanding
alternative treatment option.
Before each interview, relevant information about the patients diagnosis, presenting symptoms and recommended treatment was obtained from the case notes and discussion with the clinical team. Where a patient was prescribed more than one form of psychotropic medication, the interview focused on the medication that was judged to be the patients main treatment. This information was disclosed to the patient during the MacCATT interview (which took approximately 20 min to complete) together with standardised information about the features, benefits and risks of the particular recommended treatment (based on UK Psychiatric Pharmacy Group Information leaflets; http://www.ukppg.org.uk). The benefits and risks of no treatment were then given. All MacCATT interviews were audiotaped and transcribed.
Interrater reliability
On completion of each interview, the research interviewer (C.M. or R.C.)
made a judgement about whether the patient did or did not have capacity to
make a treatment decision. We describe this as a binary
assessment of capacity, to distinguish it from performance on the various
sub-scales of the MacCATT. This binary judgement was based on both the
MacCATT and a clinical interview with the patient and was withheld from
the other interviewer until both assessments had been made. A member of the
clinical team, usually from the nursing staff, was then asked whether they
judged the patient to have capacity to make a treatment decision. The
interviewer also scored understanding, appreciation, reasoning and expression
of choice according to MacCATT guidelines for each patient she had
interviewed.
The anonymised, typed transcripts of all the MacCATT interviews conducted by RI 1 were distributed to a panel of three consultant psychiatrists (A.S.D., M.H., G.S.) and one consultant psychologist (P.H.). Each panel member independently rated whether they judged each patient to have capacity to make a decision about their own treatment. The binary rating was based on the definition of inability to make decisions proposed in the Draft Mental Incapacity Bill (England and Wales) (Department for Constitutional Affairs, 2003) (now the Mental Capacity Act 2005). This states that persons are unable to make a decision for themselves if:
The panels training consisted of a brief discussion about using this definition to make a capacity judgement. For each case the judgement about capacity was rated as very easy, moderately easy, moderately difficult or difficult, where 1 was very easy and 4 was difficult. When the participant was judged to lack capacity, the panel member indicated in which area they had performed poorly (ad). One panel member (M.H.) also rated each typed transcript according to MacCATT criteria.
The anonymised typed transcripts from RI 2 were distributed to panel members once all the transcripts from RI 1 had been rated and returned. This time clinical information was provided with these transcripts in the form of brief summaries (about 200 words) that outlined the reason for admission, details of previous contact with psychiatric services and risk of harm to self or others. Finally, after all the transcripts had been rated, the sources of disagreement for cases in which opinion had been divided were explored in a discussion between the panel members, a lawyer with a special interest in mental capacity (G.R.), and the interviewers.
Statistical analyses
Data analysis was performed using the Statistical Package for the Social
Sciences Version 11 (SPSS,
2001) and STATA (release 8.0;
Stata Corporation, 2003).
Cohens kappa correlation coefficient and weighted kappa values (using
STATA) were calculated to examine the correlations between the different
assessments of capacity.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The sample comprised 38 men and 17 women with a mean age of 36 years (s.d.=12.4). Of these, 31 (56.3%) had the following psychotic illnesses (ICD10 F20F29; World Health Organization, 1993): schizophrenia (19), schizoaffective disorder (5) and other psychotic disorder (7). Seven patients (12.7%) had a diagnosis of bipolar affective disorder (ICD10 F31), 16 patients (29.1%) had a diagnosis of depression (ICD10 F32F33) and 1 patient (1.8%) had borderline personality disorder (ICD10 F60.3). Nineteen patients (34.5%) had been admitted involuntarily under the Mental Health Act 1983 whereas the remaining 36 had agreed to voluntary admission. There were no significant demographic differences between the group of patients that participated and the non-participants (comprising excluded and ineligible patients, those for whom only one interview was completed, and those who refused to take part) except that the latter tended to be older and there was a trend for non-participants to be female. The groups did not differ in terms of diagnosis, admission status (including type of section under the Mental Health Act 1983) or number of previous admissions. A comparison of the two groups is shown in Table 1.
|
Interrater agreements
Interrater reliability between two interviewers making separate capacity assessments, at different times, on the same patient
There was near-perfect agreement
(Landis & Koch, 1977)
between the two interviewers binary judgements of mental capacity using
two separate interviews, each based on both MacCATT and a clinical
interview, with a kappa value of 0.82. The two interviewers agreed on binary
capacity judgements in 91.0% of cases and rated 43.6% (24) and 45.5% (25) of
patients as lacking capacity, respectively.
The weighted kappa values for the MacCATT sub-scale scores from two separate interviews were as follows: understanding, 0.65; understanding alternative treatment option, 0.56; reasoning, 0.54; appreciation, 0.71; expressing a choice, 0.33. According to Landis & Kochs (1977) interpretation of kappa, this translates to a substantial level of agreement for understanding and appreciation, a moderate level for understanding the alternative treatment and reasoning, and a fair level of agreement for expressing a choice.
Interrater reliabilities between interviewers against expert clinicians, for the same interview
There was a moderate level of agreement
(Landis & Koch, 1977) for
binary capacity judgements between a panel of experts and RI 1 using typed
transcripts from the same MacCATT interviews, with a mean kappa value
of 0.60 (Table 2). However, in
line with our hypothesis, there was near-perfect agreement
(Landis & Koch, 1977) for
binary capacity judgements when brief summaries (outlining the reason for
admission, past psychiatric history and risk issues) were supplied in addition
to the typed MacCATT transcripts, with a mean kappa value of 0.84
(Table 3).
|
|
Interrater reliabilities for the MacCATT sub-scales between the interviewers and an expert clinician, based on the same interview
The level of agreement (weighted kappa values) for the individual
MacCATT subscale scores from the same interview scored by the
interviewer and a senior clinician are shown in
Table 4 for RI 1 and RI 2. For
RI 2 additional clinical information was provided. Under these conditions all
kappas were above 0.8.
|
Interrater agreement of capacity judgements about a patient between the interviewers and the clinical team
There was a moderate level of agreement
(Landis & Koch, 1977) for
binary capacity judgements between the interviewers and members of the
clinical teams responsible for the patients care (mean kappa=0.51).
Sources of disagreement between judgements
As hypothesised, the disagreement about capacity judgements was less when
the panel members were provided with additional clinical information. This is
reflected in the mean kappa values for binary capacity judgements (0.82
compared with 0.60). For the capacity ratings based on MacCATT
transcripts and clinical vignettes, there was a clear consensus (at least four
of the five raters agreed with each other) in 53 out of 55 cases. The panel
members mean difficulty rating was 2.65 (s.d.=0.21) for cases where the
judgement was split compared with 1.92 (s.d.=0.74) when the consensus was
clear. This difference was not statistically significant (t=1.39,
d.f.=53, P=0.17). For ratings based solely on MacCATT
transcripts there was a clear consensus in 48 cases.
When ratings had been completed, all raters met to discuss cases where there had been disagreement. We identified variations in the panel members interpretations of the participants reasoning and appreciation abilities to be the main source of disagreement in reaching binary capacity judgements. The less stringent view was that evidence of good reasoning at some point in the interview, with some sensible answers and some consistency with the end decision, was sufficient evidence of preserved reasoning ability. The alternative view was that anything more than trivial internal inconsistencies in the patients arguments was evidence of poor reasoning and sufficient to deem the patient incompetent. Similarly, the more lenient interpretation of patients fluctuations in the appreciation of their disorder and need for treatment was that even temporary glimpses of insight suggested they were at some level able to appreciate the relevance of this information for themselves. The more stringent view was that any significant fluctuations meant that a patients capacity was impaired. Underlying these different views was an uncertainty as to the precise degree of inconsistency in reasoning and appreciation required to establish incapacity.
Other issues were also identified. First, there was probably a bias towards judging a patient as having capacity if they made the apparently correct decision, agreeing to treatment. Second, panel members felt that for more difficult capacity judgements it would have been important to ask the patient additional questions outside the constraints of the MacCATT interview, and also to reassess the patient at another time. Finally, a difficulty arose in one case from uncertainty about whether odd use of language was attributable to the patient speaking English as a second language or to the patients psychopathology.
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Binary capacity judgements
As far as we are aware, this is the only study of the reliability of binary
capacity judgements, guided by the MacCATT and clinical judgement, from
two separate interviews of the same patient. Previous work on the reliability
of capacity assessments in mentally ill people has consisted of different
individuals rating transcripts or videos of the same interview. However, in
clinical practice we would expect much of the variation that occurs between
raters to derive from the way in which the interview itself is conducted.
There has also been more attention paid to the reliability of rating different
components of capacity (sub-scale ratings) than to the overall binary (yes/no)
judgement (Roth et al,
1977; Janofsky et al,
1992; Bean et al,
1994; Grisso et al,
1997). We would argue that the latter is more important
clinically.
Our results suggest that, in combination with a clinical interview, the MacCATT can be used to produce extremely reliable binary judgements of capacity, as currently defined, under these circumstances. The weighted kappa values for the sub-scale scores also show that the MacCATT can be used reliably by two interviewers. The greater strength of agreement seen for binary capacity judgements compared with sub-scale scores alone is understandable: the additional clinical interview used for overall capacity judgements allowed important clinical and contextual factors about the patient to be taken into account.
We also investigated the level of agreement for binary judgements of capacity using the same interview and found that a panel of senior clinicians was able to agree on this even after minimal training on the method of assessment. This is important for future research as it indicates that capacity can be reliably assessed on the basis of transcribed interviews. The level of agreement substantially improved when the panel members were provided with clinical information to aid the judgement. This is of course the context in which clinical assessments are made, and the authors of the MacCATT have not suggested that it should be used in isolation (Grisso et al, 1997). It seems most likely that the improved kappa values were a function of the increased information available to the panel but it is also possible that the experience gained from rating the first set of MacCATT transcripts may have contributed. Care was taken to prevent discussion about individuals techniques until ratings of both sets of transcripts were completed. The weighted kappa values for the subscale scores rated by the interviewer and a senior clinician also suggest that the MacCATT can be used reliably.
Strengths of the study
The consecutive sample design included patients with a range of psychiatric
diagnoses admitted both voluntarily and involuntarily and seen at an early
stage in their admission. It was therefore reasonably representative of the
heterogeneous mix of patients seen in clinical practice, ill enough to warrant
hospitalisation. In addition, the number of patients recruited and seen for
two interviews was larger than in previous studies, conferring additional
statistical power to our findings (Roth
et al, 1977; Janofsky
et al, 1992; Bean
et al, 1994; Grisso
et al, 1997; Wong
et al, 2000; Bellhouse
et al, 2003). By using Cohens kappa coefficient,
which takes account of chance agreements, we also employed a more rigorous
measure of reliability than that used in the original study of Grisso et
al (1997) describing the
interrater reliability of the MacCATT for the same interview of
psychiatric patients. In assessing agreement between two interviewers
performing separate interviews we have attempted to reflect the likely reality
of clinical practice. Our measure of agreement is effectively a hybrid of
interrater and testretest reliability, and as such we would suspect it
to yield lower kappa values than more usual judgements of interrater agreement
where the same interview is assessed.
Limitations of the study
Fifty-seven per cent of the admitted patients were not included in the
study. However, this is unlikely to limit the validity of the results unless a
significant proportion of those patients would have presented special
difficulties in the assessment of their capacity. We cannot be sure about
this, but it is unlikely to be the case since the clinical backgrounds of
these patients did not differ significantly from those of patients who did
participate.
In addition, we noticed that patients had difficulty understanding the risks and benefits of no treatment, which we used as the alternative treatment option. Similar problems have been noted in previous studies and Wong et al (2000) suggest it may be inadequate to rely on capacity assessments that involve more abstract and complex elements that are cognitively demanding and depend on sophisticated verbal expressive skills. For example, people in general find it more difficult to reason on the basis of lack of harm (or benefit) rather than positive benefits (or harm), even though they may be functionally equivalent (Kahneman & Tversky, 1984). In spite of this, a moderate level of agreement was seen between the two interviewers in this study for the understanding alternative treatment element of capacity.
We encountered some difficulties when using the MacCATT. First, as suggested elsewhere, it may be appropriate to use a staged approach, asking first for a spontaneous account of the patients existing understanding of their condition and treatment before embarking on the MacCATT interview (Wong et al, 2000). This would identify patients with a good pre-existing understanding of their condition and treatment for whom much of the disclosure part of the interview could be shortened or omitted. Some patients who clearly had capacity found the interview somewhat demeaning as they were asked, for example, to recall information when it was already clear that they could do so without difficulty.
Other patients found that an overwhelming amount of concentration was required during the disclosure of information used to test understanding in the MacCATT, to the extent that it may have constituted a memory test for some rather than assessing understanding per se. Previous studies have shown that by reducing memory load with an information sheet, in addition to a verbal disclosure, capacity can be significantly improved in some individuals (Wong et al, 2000; Bellhouse et al, 2003). This would be another possible way of tailoring the MacCATT to individual needs.
Clinical judgement of capacity
Although clinical judgements of capacity are dichotomous, we think it is
useful to view the underlying processes as a spectrum. In exploring the
differences of opinion between capacity judgements in this study, we found the
sliding scale approach, encompassing the idea of proportionality, to provide a
sensible and useful rationale for tackling this problem. This approach takes
the severity of the consequences of the task-specific decision (in this case
refusing treatment) into account and makes a judgement of incapacity more
likely as the seriousness of potential risks for the patient increases
(Gunn et al, 1999;
Wong et al, 1999;
Ms B v. An NHS Hospital Trust,
2002; Buchanan,
2004). Even with this approach, for the two cases in our sample
where opinion was divided about the patients capacity we remained
unable to reach unanimous decisions.
This study has shown that two clinicians can reliably agree about capacity to decide about treatment in the early stages of admission to a psychiatric hospital, using a combination of the MacCATT and a clinical interview. It has also shown that for research purposes a panel of senior clinicians can reliably assess capacity using transcribed interviews. Semi-structured interviews are intended to improve the reliability of capacity assessments and our results suggest that this is the case with the MacCATT interview. This reliability study has not allowed us to comment on the validity of our assessments of mental capacity. Mental capacity is a complex construct that requires consideration and assessment of a number of social and other contextual factors on an individual basis for each patient. This makes it impossible to test criterion validity of capacity assessments as there is no gold standard. The main use of the MacCATT might be to ensure that the full range of necessary abilities is considered when making a capacity judgement. We now know that in combination with a clinical interview this allows a rigorous and reliable assessment of mental capacity.
![]() |
Clinical Implications and Limitations |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
LIMITATIONS
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Bean, G., Nishisato, S., Rector, N., et al (1994) The psychometric properties of the Competency Interview Schedule. Canadian Journal of Psychiatry, 39, 368 376.[Medline]
Bellhouse, J., Holland, A. J., Clare, I. C. H., et al (2003) Capacity-based mental health legislation and its impact on clinical practice: 2) treatment in hospital. Journal of Mental Health Law, July, 24 28.
Buchanan, A. (2004) Mental capacity, legal
competence and consent to treatment. Journal of the Royal Society
of Medicine, 97, 415
420.
Department for Constitutional Affairs (2003) Draft Mental Incapacity Bill. Cmnd. 5859-I. London: Stationery Office.
Department of Health (2002) Draft Mental Health Bill. Cmnd 5538-I. London: Stationery Office.
Department of Health (2004) Draft Mental Health Bill. Cmnd. 6305-1. London: Stationery Office.
Doyal, L. (1997) Informed consent in medical
research: Journals should not publish research to which patients have not
given fully informed consent with three exceptions.
BMJ, 314, 1107
1111.
Expert Committee (1999) Review of the Mental Health Act. London: Stationery Office.
Fulford, K. W. M. & Sayce, L. (1998) Invited commentary on: Mental health legislation is now a harmful anachronism. Psychiatric Bulletin, 22, 666 670.
Grisso, T. & Appelbaum, P. S. (1995a) The MacArthur Treatment Competence Study: II Measure of abilities related to competence to consent to treatment. Law and Human Behaviour, 19, 127 148.[CrossRef]
Grisso, T. & Appelbaum, P. S. (1995b) The MacArthur Treatment Competence Study: III Abilities of patients to consent to psychiatric and medical treatments. Law and Human Behaviour, 19, 149 173.[CrossRef]
Grisso, T. & Appelbaum, P. S. (1998) Assessing Competence to Consent to Treatment: A Guide for Physicians and Other Health Professionals. New York: Oxford University Press.
Grisso, T., Appelbaum, P. S., Hill-Fotouhi, C. (1997) The MacCATT: a clinical tool to assess patients capacities to make treatment decisions. Psychiatric Services, 48, 1415 1419.[Abstract]
Gunn, M. J., Wong, J. G., Clare, I. C. H., et al (1999) Decision-making capacity. Medical Law Review, 7, 269 306.[Medline]
Gunn, M. J., Wong, J. G., Clare, I. C. H., et al (2000) Medical research and incompetent adults. Journal of Mental Health Law, February, 60 71.
Janofsky, J. S., McCarthy, R. J., Folstein, M. F. (1992) The Hopkins Competency Assessment Test: a brief method for evaluating patientscapacity to give informed consent. Hospital and Community Psychiatry, 43, 132 136.[Medline]
Kahneman, D. & Tversky, A. (1984) Choices, values and frames. American Psychologist, 39, 341 350.[CrossRef]
Kitamura, F., Tomoda, A Tsukada, K., et al (1998) Method for assessment of competency to consent in the mentally ill. Rationale, development, and comparison with the medically ill. International Journal of Law and Psychiatry, 21, 223 244.[CrossRef][Medline]
Landis, J. R. & Koch, G. G. (1977) The measurement of observer agreement for categorical data. Biometrics, 33, 159 174.[Medline]
Ms B v. An NHS Hospital Trust [2002] EWHC 429 (Fam).
Osborn, D. P. J. (1999) Research and ethics: leaving exclusion behind. Current Opinion in Psychiatry, 12, 601 604.[CrossRef][Medline]
Roth, L. H., Meisel, A. & Lidz, C. W. (1977) Tests of competency to consent to treatment. American Journal of Psychiatry, 134, 279 284.[Abstract]
SPSS (2001) SPSS Base 11.0.1 Users Guide. Chicago, IL: SPSS.
Stata Corporation (2003) Stata Release 8.0. College Station, TX: Stata.
Wong, J. G., Clare, I. C. H., Gunn, M. J., et al (1999) Capacity to make health care decisions: its importance in clinical practice. Psychological Medicine, 29, 437 446.[CrossRef][Medline]
Wong, J. G., Clare, I. C. H., Holland, A. J., et al (2000) The capacity of people with a "mental disability" to make a health care decision. Psychological Medicine, 30, 295 306.[CrossRef][Medline]
World Health Organization (1993) The ICD10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research.Geneva: WHO.
Received for publication January 24, 2005. Revision received April 11, 2005. Accepted for publication May 3, 2005.
Related articles in BJP: