Department of Psychiatry, St George's Hospital Medical School, London, UK
Correspondence: Andrea Cohen, Shaftesbury Clinic, Springfield Hospital, Glenburnie Road, London SW17 7DJ, UK. Tel: 020 8682 0033
* This paper should be read in conjunction with Cohen & Eastman
(1997). Both summarise material
presented in Cohen & Eastman
(2000).
1 Tansella & Thornicroft
(1998) also describe the
geographical dimension, which refers to different levels within
the health care system: country/region level, local level (i.e. catchment
area) and patient level.
![]() |
ABSTRACT |
---|
Aims To propose a framework for addressing the measurement of outcome in relation to mentally disordered offenders (MDOs).
Method Based on a literature search, the paper reviews the definition and measurement of outcome in general mental health care and specifically in relation to MDOs. It analyses the problems of conducting outcome research in relation to MDOs.
Results A framework for outcome measurement in relation to MDOs is presented. Outcome is placed within a broader framework that relates to service evaluation.
Conclusions Current measurement of outcome in relation to MDOs is inadequate. A comprehensive framework that acknowledges the multi-dimensional nature of outcome is essential.
Researchers must be able to justify the dimensions they prioritise.
![]() |
INTRODUCTION |
---|
![]() |
DEFINITION AND MEASUREMENT OF OUTCOME |
---|
Ovretveit (1995) defines health service outcomes as the effect on a person or population that can be attributed to a health treatment, service or intervention. However, establishing outcome in relation to mental health interventions, even in general mental health, is poorly developed. Psychiatric disorders and their associated social disabilities are complex and multi-factorial in their aetiology and manifestation (Wing et al, 1992). Baseline information is limited or non-existent, and outcomes are multi-dimensional and difficult both to define and measure. There are also difficulties in defining operationally many of the treatments and interventions available. Different treatments may be delivered to the same patient by different professionals, and multi-agency involvement adds further complexity. Consequently, demonstrating valid and reliable causal relationships between specific interventions and outcomes is problematic. Ovretveit (1995) notes that outcome measurement frequently focuses on end-points rather than health gains made during the treatment process, and is critical of the tendency for outcome measurements to fail adequately to include the effects of other services and environments, or other factors that affect health. He also bemoans the tendency of outcome studies to overlook patient views and the quality of service delivery. However, he notes how costly and methodologically difficult it is routinely to measure outcome effectively, and that commissioners who require providers to measure outcome will pay in higher prices.
![]() |
PRINCIPLES OF OUTCOME MEASUREMENT |
---|
These principles provide a comprehensive framework for outcome measurement. Although it is self-evident that most researchers will be unable to measure all of these areas or adhere strictly to all the principles advocated, the framework is still useful. It forces us to adopt a broad perspective about outcome measurement and to recognise the limitations on what is achievable. It also forces us to acknowledge that we may sometimes be prioritising only one dimension of outcome (and a small one at that), neglecting other domains in its favour. Public policy, values and resources will partly drive what aspect of outcome is prioritised and measured, as well as the methods used to achieve this. Hence, there must be explicit acknowledgement of what is not being measured and why we should be able to justify, for example, why it is more important to measure recidivism as an outcome rather than symptom reduction or quality of life.
![]() |
PROBLEMS WITH MEASURING OUTCOME IN RELATION TO MDOs |
---|
![]() |
OUTCOME, QUALITY AND SERVICE EVALUATION |
---|
A similar conceptualisation is offered by Donabedian (1980). He divides research about the quality of health care into studies that address structures (e.g. provider systems, organisation of systems, characteristics of treating facilities), process (specific clinical interventions) and outcome.
Berwick (1989) outlines four types of health services research that relate to quality of care:
According to Atkisson et al (1992), progress with the paradigms presented by Donabedian (1980) and Berwick (1989) is required in order to advance research about quality of care.
Glover & Kamis-Gould (1996) propose a model of service evaluation that covers two broad aspects of an organisation/system. The first relates to the capacity of the system. Capacity variables include human and financial resources, the range and quality of clinical facilities and the technical capacity to operate, coordinate and monitor all aspects of organisational functioning. The second relates to the performance of the system. This is concerned with responsiveness and accessibility (e.g. congruence with local needs, cultural sensitivity, promptness and sensitivity of response to clients). Performance is also measured in terms of efficient use of resources (i.e. levels of productivity, cost containment, occupancy rates) and effectiveness.
According to Jenkins (1990), aspects of service provision that can be most easily measured at present tend to be those that relate to service input and resources rather than to service outcome. She notes that input is relatively straightforward to measure, and that process tends to be measured in terms of performance or activity indicators (e.g. occupied bed-days). Process indicators related to delivery of specific interventions or the nature of therapeutic relationships are more difficult to measure and are unlikely to be available routinely. Jenkins also points out that process indicators are frequently selected on the basis of what is collectable (or already available), rather than being derived from previously specified key aspects of performance. Indeed, although there may be good ad hoc studies relevant to some desirable process measures, there is, in fact, a profound lack of ongoing data that could be of use in the monitoring process and, in particular, in monitoring the meeting of mental health needs. Jenkins notes that the measurement of outcomes is more complex than the measurement of input and process. She points out that input and process indicators are often used as proxy measures of outcome, which she suggests is based on faulty logic that is, that service utilisation (process indicator) is equal to improvement (outcome). So, just as service utilisation is a poor proxy for need (Cohen & Eastman, 1997), so too is it a poor proxy for outcome.
![]() |
INPUT, PROCESS AND OUTCOME INDICATORS IN FORENSIC MENTAL HEALTH |
---|
As a policy starting point, Jenkins offers a series of health objectives specific to MDOs. These are essentially policy objectives and are clearly influenced by Health of the Nation (Department of Health, 1992) targets. Examples include:
She goes on to suggest a range of input, process and outcome indicators that relate to her proposed objectives. Input indicators include:
She then argues that process indicators should be established that reflect activity on all the above input indicators.
Finally, Jenkins identifies a number of outcome indicators:
Jenkins' lists of objectives and indicators may no longer accurately reflect policy priorities of the current government, and it is important to note their historical limitations. They were formulated when diversion from the criminal justice system was particularly high on the political agenda and before the publication of the Reed Committee Report (Department of Health/Home Office, 1992). Although the Reed Report itself then reinforced the need for diversion from the criminal justice system, it also suggested an additional range of objectives and indicators, such as systems to identify and treat patients who no longer require particular levels of security. Similarly, the recent Ashworth Inquiry (Fallon et al, 1999), and the wealth of national and local inquiries following homicide by people with mental illness, have subsequently suggested a wide range of other potential objectives and indicators (e.g. Sheppard, 1996; National Confidential Inquiry into Suicide and Homicide by People with Mental Illness, 1999). Hence, a current set of input, process and outcome indicators might now be drawn significantly differently. It is important to recognise, therefore, that appropriate objectives and indicators will change over time, according to altered policy considerations, as well as in response to changes in service structure and advances in the ability to measure need and outcome.
There are, in any event, a number of criticisms that can be levelled at Jenkins' earlier choice of objectives and indicators. One of their disadvantages is that they reflect a public health stance that tends to neglect outcome at the individual level. Jenkins' system also includes some objectives that are difficult to conceptualise as being legitimate objectives of psychiatric services and are dependent upon many factors that are arguably beyond the remit of MDO health or even social service interventions (e.g. to reduce homelessness). Further, the majority of indicators suggested have no adequate baselines specific to MDOs and are not routinely measured at a local, regional or national level, and it is difficult to envisage how many of them could be measured reliably and validly in the future, particularly at a level that would be useful to commissioners and service providers. It is also noteworthy that many of Jenkins' suggested outcome indicators fall well short of being direct measures of outcome. For example, both the number of patients detained under Part III of the Mental Health Act and readmission rates represent indirect or proxy measures of outcome, with an assumption that, in relation to the achievement of goals relating to each, good will follow (e.g. that service utilisation equates with a positive outcome). Indeed, of Jenkins' outcome indicators, only the suicide rate in prison and standardised mortality ratios can be seen as direct outcome variables, although these indicators are not currently statistically available specifically in relation to MDOs, and the extent to which they directly reflect mental health outcome is also debatable.
The criticisms levelled at Jenkins' system, which at face value appear entirely reasonable, illustrate just how much of a challenge it is to attempt to formulate any system. It is very difficult to select objectives and indicators that are both reasonable and realistic (e.g. measurable in relation to baselines, input, process and outcome) and that take into account broader policy objectives, as well as clinical and system realities.
![]() |
CONCEPTUAL FRAMEWORK FOR EVALUATING FORENSIC MENTAL HEALTH SERVICES |
---|
|
The conceptual model of outcome measurement and service evaluation presented in Table 1 poses considerable challenges in both methodological and practical terms. It is therefore unlikely that we shall see anything that approaches the degree of comprehensiveness suggested by the model in the near future, although this should be the gold standard towards which to strive. The existing mechanisms and measures available both for estimating outcome and for evaluating services in relation to MDOs verge on being hopelessly inadequate. At the root of this problem is a lack of knowledge about how particular clinical interventions and services influence outcome. Indeed, there are not even any generally agreed upon service designs and protocols that might be measured in their effects. This inhibits not only the determination of appropriate outcome measures but also the definition of need itself. How can we define need if there is little agreement over the details of effective service response to need? Until we are able adequately to answer questions about input, process and outcome, we shall not be able properly to answer questions about ability to benefit and, hence, about need.
![]() |
Clinical Implications and Limitations |
---|
LIMITATIONS
![]() |
REFERENCES |
---|
Bailey, J. & MacCulloch, M. J. (1992) Patterns of reconviction in patients discharged directly to the community from a Special Hospital: implications for aftercare. Journal of Forensic Psychiatry, 3, 445-462.
Beecham, J. & Chisholm, D. (1995) Mental health economics. In Measurement for Mental Health: Contributions from The College Research Unit (ed. J. K. Wing) pp. 55-65. London: College Research Unit, Royal College of Psychiatrists.
Berwick, D. M. (1989) Health service research and quality of care: assignments for the 1990s. Medical Care, 27, 763-771.[Medline]
Blumenthal, S. & Lavender, T. (2000) Violence and Mental Disorder: A Critical Aid to the Assessment and Management of Risk. Hay-on-Wye: Zito Trust.
Brugha, T. S. & Lindsay, F. (1996) Quality of mental health service care: the forgotten pathway from process to outcome. In Mental Health Outcome Indicators (eds G. Thornicroft & M. Tansella) pp. 93-109. Germany: Springer-Verlag.
Buchanan, A. (1998) Criminal conviction after discharge from special (high security) hospital. Incidence in the first 10 years. British Journal of Psychiatry, 172, 472-476.[Abstract]
Cohen, A. & Eastman, N. (1997) Needs assessment for mentally disordered offenders and others requiring similar services. Theoretical issues and a methodological framework. British Journal of Psychiatry, 171, 412-416.[Abstract]
Cohen, A. & Eastman, N. (2000) Assessing Forensic Mental Health Needs: Policy, Theory and Research. London: Gaskell.
Department of Health (1992) The Health of the Nation. Cm. 1986. London: HMSO.
Department of Health/Home Office (1992) Review of Health and Social Services for Mentally Disordered Offenders and Others Requiring Similar Services: Final Summary Report (Reed report). Cm 2088. London: HMSO.
Donabedian, A. (1980) Explorations in Quality Assessment and Monitoring: The Definition of Quality and Approaches to its Assessment. Ann Arbor, MI: Health Administration Press.
Eastman, N. (1999) Public health psychiatry or
crime prevention? British Medical Journal,
318,
549-551.
Fallon, P., Bluglass, R., Edwards, B., et al (1999) Report of the Committee of Inquiry into the Personality Disorder Unit, Ashworth Special Hospital. Cm. 4195. London: HMSO.
Glover, G. & Kamis-Gould, E. (1996) Performance indicators in mental health services. In Commissioning Mental Health Services (eds G. Thornicroft & G. Strathdee), pp. 265-272. London: HMSO.
Home Office/Department of Health (1999) Managing Dangerous People with Severe Personality Disorder: Proposals for Policy Development. London: Home Office.
Jenkins, R. (1990) Towards a system of outcome indicators for mental health care. British Journal of Psychiatry, 157, 500-514.[Abstract]
National Confidential Inquiry into Suicide and Homicide by People with Mental Illness (1999) Safer Services: National Confidential Inquiry into Suicide and Homicide by People with Mental Illness. London: Department of Health.
National Health Service Management Executive (1991) Assessing Health Care Needs: A DHA Discussion Paper. London: Department of Health.
Ovretveit, J. (1995) Purchasing for Health. Buckingham: Open University Press.
Robertson, G. (1997) Research in forensic psychiatry. Journal of Forensic Psychiatry, 8, 501-503.
Sheppard, D. (1996) Learning the Lessons: Mental Health Inquiry Reports Published in England and Wales between 1969 and 1996 and their Recommendations for Improving Practice (2nd edn). London: Zito Trust.
Tansella, M. & Thornicroft, G. (1998) A conceptual framework for mental health services: the matrix model. Psychological Medicine, 28, 503-508.[CrossRef][Medline]
Wing, J., Brewin, C. R. & Thornicroft, G. (1992) Defining mental health needs. In Measuring Mental Health Needs (eds G. Thornicroft, C. R. Brewin & J. Wing), pp. 1-17. London: Gaskell.
Received for publication December 13, 1999. Revision received June 5, 2000. Accepted for publication June 9, 2000.
HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
Psychiatric Bulletin | Advances in Psychiatric Treatment | All RCPsych Journals |