NHS Centre for Reviews and Dissemination, University of York, York
Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, Leeds
Department of Health Studies, University of York, York
Correspondence: Dr S. M. Gilbody, Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, 15 Hyde Terrace, Leeds LS2 9LT, UK. Tel: 0113 2331899; Fax: 0113 2433719; e-mail: s.m.gilbody{at}leeds.ac.uk
Governmental policy statements on mental health practice over the past decade have emphasised the importance of routinely measuring individual patient outcomes (Department of Health, 1991, 1998; Secretary of State for Health, 1999). Despite the availability of various standardised tools with which to measure the symptom severity of common psychiatric disorders, patient need and wider quality of life and health status, little is known about the actual use of standardised outcomes measures by clinicians (Slade et al, 1999). One previous survey of 73 consultant psychiatrists from 1989 established which of a pre-specified range of symptom-based clinical measures were in use at that time (Rice & Donnelly, 1992). In view of the central role given to outcomes measurement in the recent National Service Framework for Mental Health (Secretary of State for Health, 1999) we decided to undertake a survey of the current use of outcomes measures in psychiatric practice in the UK.
NATIONAL SURVEY
Questionnaire design
We constructed and piloted a self-completion/self-report questionnaire that
sought to identify the routine standardised outcomes measures used by adult
psychiatrists for the purpose of:
Respondents were asked about the use of outcomes measures for the following problems: depression, anxiety and related disorders; schizophrenia and other psychoses; cognitive impairment; drugs and alcohol. Respondents also were asked what outcomes measures their trusts routinely collected or asked them to collect.
Survey method
We conducted a postal questionnaire survey between July and September 2000
of 500 consultant psychiatrists practising in the UK National Health Service
(NHS) chosen randomly from UK NHS clinicians responsible for the psychiatric
care of working-age adults listed in the Medical Directory.
Questionnaire responses
In total, 369 (74%) of the 500 questionaires were returned, of which 29
were incomplete or ineligible (raw eligible response rate 340/500, 68%;
modified response rate 340/471, 72%). Most were general psychiatrists (82%)
and worked in non-teaching hospital settings (66%). Details of their responses
are given in Table 1.
|
Case identification and severity measurement
Depression/anxiety and cognitive impairment were the disorders where
outcomes measures were used most commonly for identifying and assessing the
severity of psychiatric disorders, with around half of clinicians using these
measures either routinely or occasionally. The most commonly used measures
were the Beck Depression Inventory (Beck
& Ward, 1961) (61/340), the Hospital Anxiety and Depression
Scale (Zigmond & Snaith,
1983) (53/340) and the Hamilton Rating Scale for Depression
(Hamilton, 1967) (46/340).
The most commonly used measure in detecting cognitive impairment was the Mini-Mental State Examination (Folstein et al, 1975). For disorders such as schizophrenia, the majority of consultants (72.9%; 95% CI 67.9-77.6) never used standardised measures. For drug and alcohol problems, the most commonly reported measure was the CAGE questionnaire (Mayfield et al, 1974) (10/340).
Assessing social functioning, quality of life and needs
Only a small minority used a standardised questionnaire for detecting
patients' needs and deficits in social functioning and quality of life. The
most commonly used was the Health of the Nation Outcome Scale (HoNOS;
Curtis & Beevor, 1995).
Specific needs assessment tools, such as the Medical Research
Council (MRC) Needs for Care and the Camberwell Assessment of Needs, rarely
were reported as being used in routine practice.
Measuring clinical change over time
Standardised measures most commonly were used in order to measure change
over time for depression and anxiety problems, with 19.4% (95% CI 15.3-24.0)
of consultants using them either routinely or occasionally. The most commonly
reported measures were the Beck Depression Inventory (49/340); the Hospital
Anxiety and Depression Scale (41/340) and the Hamilton Rating Scale for
Depression (23/340). For those with dementia, the most commonly used
questionnaire was the Mini-Mental State Examination (60/340). For other
conditions, clinicians rarely used standardised measures.
Clinical audit
Overall, standardised questionnaires were used even less for clinical audit
than for the other purposes outlined above. The most commonly reported
condition for which they were used was depression/anxiety, where the Beck
Depression Inventory was the most common instrument; for other conditions
(schizophrenia; drugs and alcohol; and dementia) the HoNOS was used.
Outcomes measures routinely collected by hospitals/trusts
Very few clinicians (46/340, 13.5%; 95% CI 10.0-17.6) reported being
required routinely to collect standardised outcomes measures for all patients
by their trust. When asked specifically about the HoNOS, 88/340 (26%; 95% CI
21.3-30.1) reported being asked to collect these data for certain patients,
and only 28/340 (8.2%; 95% CI 5.5-11.7) reported using specific needs
assessment tools (such as the Camberwell Assessment of Need and the MRC Needs
for Care).
In contrast to standardised questionnaires, such as the HoNOS, trusts commonly collected routine administrative data, such as length of stay (86.2%; 95% CI 82.7-89.9) and readmission (70.6%; 95% CI 65.4-75.4).
General comments relating to the use and experience of outcomes
measurement
Approximately one-third (120/340) used the questionnaire to give comments.
Forty respondents expressed a negative view of standardised outcomes measures,
questioning their ability to capture the subtlety of multi-faceted outcome and
to describe the individual patient. Terms such as "simplistic" and
"pseudo-scientific gloss" were used. One respondent expressed
reservation about any scale that "divides continuous fluctuating process
into arbitrary categories". Twenty-eight respondents explicitly
questioned the basic psychometric properties of validity, reliability and
sensitivity to change for available measures.
Respondents stated that the valid and reproducible use of outcome measures requires a robust infrastructure, particularly in terms of administration and information technology resources, to support the process (n=20). Respondents generally felt that these features were lacking and that this represented a barrier to their use. Additionally, 29 respondents felt that more time and resources would be needed if outcome measurement were to be carried out and used routinely. Twenty-two respondents stated that they did not find the results of standardised outcomes measures particularly useful in clinical practice. One respondent stated that they were more "research tools" rather than instruments that are useful in clinical practice. Another stated that the "use of scales detracts from the therapeutic relationship".
Comments were largely critical of the HoNOS (n=21) and related to: time to complete (n=16); inadequate psychometric properties (n=8); the lack of value added to routine clinical assessment (n=5); and the lack of enthusiasm among staff (n=7). Positive comments (n=7) included the fact that it could be completed by non-clinicians (n=4) and that it acted as a useful aide mémoire in clinical decision-making (n=3). One person stated that "the HoNOS, although scientifically flawed, is useful for bringing together all members of the multi-disciplinary team".
IMPLICATIONS FOR MENTAL HEALTH PRACTICE, RESEARCH AND POLICY
Outcomes measurement forms a central component of the National Service Framework, with, for example, the stipulation of a minimum data set (Glover et al, 1997), which includes the HoNOS, to be collected alongside care planning for all of those with severe mental illness and a number of outcomes indicators to be implemented on a routine basis (Secretary of State for Health, 1999). The results of this survey, in particular the barriers identified to outcomes measurement, will be of particular interest to those charged with implementing the National Service Framework for Mental Health. Several areas are worthy of further discussion.
Our main finding is that the majority of psychiatrists do not use outcomes measures in their day-to-day practice. Patient needs and psychosocial problems are measured infrequently in any standardised or consistent way, despite explicit Government policy (Glover et al, 1997; Secretary of State for Health, 1999) to adopt measures such as the HoNOS and needs assessment tools. This may reflect a wider indifference towards and failure to address psychosocial outcomes and needs, or simply a failure to use standardised measures to assess them.
The HoNOS does seem to have found a place in measuring outcome in UK mental health services, albeit a small one. Initial hopes that it would be used on a servicewide basis in order to measure outcome, define population needs and to judge the performance of individual trusts and teams (Wing, 1994) have, however, clearly not been realised. Specific concerns expressed by clinicians relating to the time taken to complete the instrument and the poor psychometric properties will need to be addressed if its use is going to be increased. On the other hand, positive aspects of outcomes measures were identified that may be useful in deciding their place in mental health services. In particular, the comments that non-medical personnel can complete instruments such as the HoNOS and that it encouraged discussion within the wider clinical team are worthy of note.
Administrative measures collected by trust such as length of stay and readmission rates although easier to collect, may bear little relation to the psychosocial outcome of the individual patient or clinical population. It is administrative outcomes that will form the basis of the performance elements of the NHS (Secretary of State for Health, 1999). Of particular concern is that these figures are the easiest to manipulate or improve, without conferring any overall health gain on the population or service under consideration (Nutley & Smith, 1998).
Teams need to be resourced adequately to collect outcomes data, and the instruments that are used must be reasonably concise and easily administered (Slade et al, 1999). Additionally, the reluctance among clinicians to collect data on a routine basis may reflect the fact that they see little benefit to themselves or their patients' care. Outcomes measurement is a technology that has opportunity costs and therefore must be shown to be beneficial in improving the quality of care as measured by actual outcomes of patients or communities. No direct evidence exists that there is a benefit in this respect for those working and being cared for in either primary care or specialist psychiatric services (Gilbody et al, 2001a,b). Successful implementation of outcomes management will need to overcome the barriers that we identified in this survey of current UK psychiatric practice.
ACKNOWLEDGMENTS
S.M.G. was supported by the UK Medical Research Council Health Services Research Training Fellowship Programme. The NHS Centre for Reviews and Dissemination and the Department of Health Studies are part of the Medical Research Council Health Services Research Collaboration (MRC HSRC).
REFERENCES
Beck, A. T. & Ward, C. H. (1961) An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571.
Curtis, R. & Beevor, A. (1995) Health of the Nation Outcome Scales (HoNOS). In Measurement for Mental Health: Contributions from the College Research Unit (ed. J. Wing), pp. 33-46. London: Royal College of Psychiatrists' Research Unit.
Department of Health (1991) The Health of the Nation: A Strategy for England. London: HMSO.
Department of Health (1998) Modernising Mental Health Services: Safe, Sound and Supportive. London: HMSO.
Folstein, M. F., Folstein, S. E. & McHugh, P. R. (1975) Mini-Mental State: a practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189-198.[CrossRef][Medline]
Gilbody, S. M., House, A. O. & Sheldon, T. A. (2001a) Outcome and needs assessment for schizophrenia and related disorders (Cochrane Review). In The Cochrane Library Issue I. Oxford: Update Software.
Gilbody, S. M., House & A. O. & Sheldon, T. A.
(2001b) Routinely administered questionnaires for
depression and anxiety: a systematic review. BMJ,
322,
406-409.
Glover, G., Knight, S., Melzer, D., et al (1997) The development of a new minimum data set for specialist mental health care. Health Trends, 29, 48-51.
Hamilton, M. (1967) Development of a rating scale for primary depressive illness. British Journal of Social and Clinical Psychology, 6, 278-296.
Mayfield, D., Millard, G. & Hall, P. (1974) The CAGE questionnaire. American Journal of Psychiatry, 131, 1121-1123.[Medline]
Nutley, S. & Smith, P. (1998) League tables for performance improvement in health care. Journal of Health Services Research and Policy, 3, 50-57.[Medline]
Rice, K. & Donnelly, P. (1992) Use of rating scales by consultant psychiatrists. Psychiatric Bulletin, 16, 329-330.
Secretary of State for Health (1999) National Service Framework Mental Health. London: HMSO.
Slade, M., Thornicroft, G. & Glover, G. S. O. (1999) The feasibility of routine outcome measures in mental health. Social Psychiatry and Psychiatric Epidemiology, 34, 243-249.[CrossRef][Medline]
Wing, J. (1994) Measuring mental health outcomes: a perspective from the Royal College of Psychiatrists. In Outcomes into Clinical Practice (ed. T. Delamonthe), pp. 147-152. London: BMJ Publishing.
Zigmond, A. S. & Snaith, R. P. (1983) The Hospital Anxiety and Depression scale. Acta Psychiatrica Scandinavica, 67, 361-370.[Medline]
Received for publication May 15, 2001. Revision received September 4, 2001. Accepted for publication October 5, 2001.
Related articles in BJP: