National Children's Bureau, 8 Wakley Street, London ECIV 7QE
* This article is based on the Rutter Lecture, delivered on 22 April 1999, at
the Faculty of Child and Adolescent Psychiatry Annual Meeting in
Manchester.
See invited commentary, p.
420, this issue.
![]() |
ABSTRACT |
---|
Aims To consider critically the use of the term evidence, to suggest limits to the value of conventionally defined evidence and to point to possible ways forward to bridge the gap between research findings and clinical practice.
Method A review of the literature relating to the use of evidence-based methods.
Results The term evidence needs to be used more widely than is conventionally the case. Substantial evidence exists from controlled trials, but there are barriers to its use.
Conclusions A move away from nonvalidated methods of intervention is both desirable and feasible. The use of qualitative methods of enquiry, both in situations where controlled trials are unlikely to be feasible and as adjuncts to quantitative methods, should be considered more seriously.
![]() |
INTRODUCTION |
---|
![]() |
EVIDENCE-BASED CHILD PSYCHIATRY |
---|
Most consultant child and adolescent psychiatrists know, of course, that evidence derived from RCTs represents the gold standard for evidence. However, in ordinary common parlance they are likely to define evidence as any information useful in making a clinical decision. Does this include experiences with similar patients? Most psychiatrists would indeed think of several relevant clinical experiences as reasonably strong evidence.
Further relevant information or evidence may be derived from the individual case about which a decision has to be made. If, during an assessment of a seven-year-old boy with the features of attention-deficit hyperactivity disorder, it becomes clear that there is a marked improvement in behaviour when it has been indicated to his parents that they need to be firmer in their control of him, it may be reasonable to embark on a course of parent counselling or parent management training without initially using stimulant medication (a scientifically substantiated treatment for this condition), at least until the effect of a psychological approach has been assessed.
Some resolution of this issue concerning the nature of evidence may be achieved if one moves away from the sharp distinction proposed by Sackett et al (1997) between scientific evidence and clinical expertise. When faced with a clinical problem, one can use information or evidence derived from the systematic study of groups of similar cases. One can also use information or evidence derived from previous experience of similar cases and from the assessment of the unique child and family that one is seeing. Rather than making a distinction between scientific evidence and clinical expertise, it might be better to think first of external evidence and then of case evidence. Case evidence refers to information derived from the unique features of the child and family in question. Bringing these two types of evidence together might help to solve the problem of what Szatmari (1998) has called "the two solitudes - the (different) spaces in which parents and professionals live, (which are) divided by the term evidence".
Both types of evidence are relevant, and the relationship of data to theory is similar, if not identical. In clinical research one moves inductively from observations of groups of similar cases to theory, which one then attempts to test using hypothetico-deductive methods, hoping to achieve generalisable findings. In the individual case, one moves inductively from the data obtained in an assessment, combining these with information derived from findings from generalisable research, to a theory about this particular case. One then tries, using a hypothetico-deductive model, to test this theory by checking it against other information or by attempting an intervention and observing to see whether or not the result is consistent with the theory. These are, or should be, interlocking processes in which evidence from unique cases feeds into generalisable research activity, and vice versa.
![]() |
LIMITATIONS OF THE RCT |
---|
![]() |
EXISTING FINDINGS FROM WELL-CONDUCTED RCTs |
---|
It is possible to draw some general conclusions from the findings of these studies. First, different conditions respond to different therapies. There is no longer any room for clinicians who are guided by only one theory and who are only prepared to apply one type of treatment to all problems they see. Second, although effect sizes are larger in so-called laboratory studies with volunteer samples (Weisz et al, 1995), they are relatively small, in the region of 0.2-0.3, even in the best outcomes obtained in clinical samples. Third, when either might have been applied, the evidence for the effectiveness of behavioural therapies is greater than the evidence for non-behavioural therapies, including non-behavioural family therapy, individual psychodynamic psychotherapy and counselling. This may be because non-behavioural therapies have been less well evaluated, and of course unevaluated therapies are not necessarily ineffective.
The implications of this finding of the superiority of evidence for behavioural therapies are, as I see them, threefold. First, in situations where behavioural or non-behavioural types of therapy might be used, and the superiority of the former has been established, these should be preferred. The second implication is that there is a need for more outcome research with non-behavioural methods, using techniques that, if found to be effective, could be applied realistically in National Health Service settings. There is little point in the conduct of research on treatments that are either prohibitively expensive or demand unrealistic levels of compliance from family members. Third, where the efficacy of behavioural treatments has been established, non-behavioural treatments should not be applied except as part of well-designed research studies examining outcome using standardised measures.
![]() |
BARRIERS TO IMPLEMENTATION OF EXISTING FINDINGS FROM TREATMENT RESEARCH |
---|
First, although our classification system is as good as one might hope for, given the limited knowledge on which it is based, it is still not strong enough to allow diagnosis to be more than a very rough guide to intervention. The motivation of the child and other family members to receive help, the presence of mental illness or personality problems in one or both parents, the social circumstances in which the family is living, the resources available and many other considerations may all be more powerful determinants of treatment than diagnosis, and together they are almost bound to be so.
Second, psychiatrists, psychologists, social workers and psychotherapists have inadequate access to the best available evidence. The textbooks do not provide it. Published reviews are often out of date, even at the time they appear. The FOCUS initiative is beginning to fill the knowledge gap with evidence-based briefings (Royal College of Psychiatrists' Research Unit, 1999), but the gap will continue to exist for many years. In addition, one may expect more up-to-date and thoroughly researched information to appear on Internet websites, such as that on evidence-based mental health.
Third, and perhaps most importantly, the resources to provide the forms of treatment best supported by evidence are lacking. There is a lack of professionals trained in evidence-based therapies because there is no one competent to supervise them. This is particularly the case for cognitive-behavioural therapy. Trainees in child and adolescent psychiatry still have too little supervised experience in the most effective therapies. The situation is changing, but many trainees still get little opportunity to develop skills in the use of medication and behavioural therapies.
Fourth, given that inevitably some consultants have been trained in forms of therapy less effective than others that have since emerged, there is a lack of opportunity to re-train under supervision. One- or two- day workshops are unlikely to be adequate, and longer attachments may be required during which consultants can observe therapies unfamiliar to them and, at the very least, undertake therapy using role-play until it is clear that they have mastered the necessary skills.
Fifth, there is the problem of the institutionalisation of particular types of therapy, especially psychoanalysis and family therapy, but one can also see similar trends developing in some forms of behavioural, cognitive-behavioural and problem-solving therapies. The whole apparatus of training institutions - their hierarchical structures, training committees, their certificates of attendance, diplomas, even degrees, their stringent admission and exclusion criteria and the financial investment that trainees have to make to become accredited - creates a system permeated by beliefs, attitudes and values that are extraordinarily impervious to new ideas and evidence that does not fit with the received dogma. There are at least a few forward-thinking, authoritative psychoanalysts who agree with this view (Fonagy, 1999).
![]() |
WHEN EXTERNAL SCIENTIFIC EVIDENCE DOES NOT EXIST |
---|
A second example arises when a child is showing an unusual type of problem such as a gender identity disorder for which RCTs do not exist. In these circumstances, as Muir Gray (1997) suggests, it will be reasonable to use information derived from specialist clinics. Third, child mental health professionals spend a great deal of their time "managing the context" (S. Kingsbury, personal communication, 1999). A visit or a telephone call to a school to discuss ways in which exclusion may be avoided, putting a lone mother in touch with a self-help group and arranging after-school activities for a boy entangled in a delinquent subgroup would all fall into this category.
![]() |
QUALITATIVE RESEARCH METHODS |
---|
What is qualitative research? Most of those who have defined qualitative research have done so by contrasting it with quantitative research. However, as we shall see, many of those engaged in qualitative research (e.g. Hammersley, 1992) do not see the two types of research as categorically different. They see both types of methodology as part of a common endeavour to try to expand knowledge through disciplined inquiry.
First, although Strauss & Corbin (1998) define qualitative research as "any type of research that produces findings not arrived at by statistical procedures or other means of quantification", this definition is not widely accepted by those engaged in qualitative work, who virtually all give the number of subjects they have investigated, and often in their conclusions describe certain explanations as more or less likely, or certain occurrences as frequent or infrequent.
A second distinction is that those engaged in qualitative research proceed by inductive methods, that is by moving from observable data to theory, whereas quantitative researchers proceed by deduction, testing theory by experiment and observation. As Medawar (1984) pointed out, those engaged in scientific, biological research are constantly moving from observation and experiment to theory, and back again.
Those engaged in qualitative research are said to focus on events in natural settings, whereas quantitative research is undertaken in experimental settings. This begs several questions. Is a child psychiatric clinic a natural setting or an experimental one? If a social scientist goes into schools and observes quantifiable behaviour, as Rutter et al (1979) have done, are they carrying out work that is experimental or natural?
Qualitative research is said to be strong on validity, and quantitative research strong on reliability. This distinction seems unnecessarily derogatory to both approaches. Surely all disciplined inquiry must involve observations that are repeatable (reliable), and use valid methods that produce data relevant to the purpose for which they were designed. The findings of qualitative research need to be reliable if they are to be generalisable, and findings from quantitative work need to be valid if they are to be of any use at all.
Qualitative research is said to be committed to an idealist epistemology and quantitative researchers committed to a view that the world can be described realistically. Again, this distinction is artificial. Picasso is quoted as saying "Anything that can be imagined is real", and quantitative researchers who think they have discovered real reality or truth are usually merely blind to historical, cross-cultural and other evidence of the social construction of truth.
Having criticised the idea that these two types of research are categorically different, it must be allowed that those engaged purely in qualitative research do, in general, have different areas of interest, use different methods and achieve different types of outcome or results when compared with those engaged in purely quantitative work. But, more and more, those engaged in relevant research use both approaches. Most problems need to be addressed by both types of method, and increasingly a mix of methods is employed (Brannen, 1992).
![]() |
QUALITATIVE RESEARCH: QUALITY CONTROL |
---|
The research method needs to be described clearly and fully so that anyone wishing to replicate the study must be able to do so on the basis of published material. If a research worker wishes to establish a finding, enough subjects must be interviewed or observed to make it unlikely that investigation of more subjects would produce conflicting evidence. The data, often audio- or video-taped, should be available for checking by an independent observer. Finally, any arguments that go against the conclusions should be considered seriously and discussed; in particular, deviant cases that go against general conclusions need to be taken into account.
![]() |
THE USE OF QUALITATIVE METHODS |
---|
There are several examples where a mix of qualitative and quantitative methods has produced useful information. Skuse et al (1998), in a quantitative study, identified family violence as an important predictor of later perpetrating behaviour in sexually abused boys. They were greatly helped to formulate their hypotheses by systematically conducted qualitative assessments carried out by psychoanalytical psychotherapists. Jones & Ramchandi (1999) have described how a series of studies, funded by the Department of Health, using a mixture of qualitative and quantitative methods has provided useful information to guide interventions where there is a suspicion of child sexual abuse. Finally, qualitative work could help to clarify another area in which there is a substantial amount of cognitive dissonance. What are reasonable and what are unreasonable expectations of the effectiveness of a child psychiatric service? Exploration of this issue using qualitative methods with managers, mental health professionals and those working in other agencies, such as social work and education, could be useful in reducing such dissonance.
Such work might enable us to reduce expectations to a more realistic level, especially in severely disturbed children. Interventions should be valued if they enable those working in different agencies to achieve common understanding of problems and similar expectations of their development and agreement on a realistic plan for the future (Wiener et al, 1999). Good inter-agency collaboration by no means necessarily brings with it better outcomes for children and families, as the Fort Bragg study showed (Bickman, 1996), but such collaboration does improve satisfaction both in families and in professional staff, and this in itself is a worthwhile achievement.
Behaviour and emotional problems are inevitable. They cannot be cured: they may be helped to improve, although perhaps only temporarily: most of our treatments achieve only marginal benefit and some do not work at all. To paraphrase part of a recent editorial in the British Medical Journal (Smith, 1999), the best child and adolescent mental health service "...will not be one that provides everything for everybody, but rather one that determines how much it wants to spend on such services, and then provides explicitly limited evidence-based services in a humane and open way without asking the impossible of its staff."
![]() |
Clinical Implications and Limitations |
---|
LIMITATIONS
![]() |
ACKNOWLEDGMENTS |
---|
![]() |
REFERENCES |
---|
Audit Commission (1999) Children in Mind: Child and Adolescent Mental Health Services. London: Audit Commission.
Barnes, J., Stein, A. & Rosenberg, W.
(1999) Evidence based medicine and evaluation of mental
health services: methodological issues and future directions.
Archives of Disease in Childhood,
80,
280-285.
Beelman, A., Pfingset, U. & Losel, F. (1994) Effects of training social competence in children: a meta-analysis of recent evaluation studies. Journal of Clinical Child Psychology, 23, 260-271.
Bickman, L. (1996) A continuum of care. More is not always better. American Psychologist, 51, 689-701.[CrossRef][Medline]
Blackwell, B. & Currah, J. (1973) The psychopharmacology of nocturnal enuresis. In Bladder Control and Enuresis (eds I. Kolvin, R. MacKeith & S. R. Meadow), pp. 231-257. London: Heinemann.
Brannen, J. (1992) Combining qualitative and quantitative approaches: an overview. In Mixing Methods: Qualitative and Quantitative Research (ed. J. Brannen), pp. 3-37. Aldershot: Avebury.
Cronbach, L. & Suppes, P. (1969) Disciplined Enquiry for Education, p. 18. London: Macmillan.
DeVaugh-Geiss, J., Maroz, G., Biederman, J., et al (1992) Clomipramine hydrochloride in childhood and adolescent obsessive-compulsive disorder - a multicenter trial. Journal of the American Academy of Child and Adolescent Psychiatry, 31, 45-49.[Medline]
Dische, S., Yule, W., Corbett, J., et al (1983) Childhood nocturnal enuresis: factors associated with outcome of treatment with an enuretic alarm. Developmental Medicine and Child Neurology, 25, 67-80.[Medline]
Emslie, G. J., Rush, A. J., Weinberg, W. A., et al (1997) A double-blind, randomized, placebo-controlled trial of fluoxetine in children and adolescents with depression. Archives of General Psychiatry, 54, 1031-1037.[Abstract]
Fombonne, E. (1998) Interpersonal therapy for adolescent depression. Child Psychology and Psychiatry Review, 3, 169-175.[CrossRef]
Fonagy, P. (1999) Reflections on psychoanalytic research problems - an Anglo-Saxon view. In An Open Door Review of Outcome Studies in Psychoanalysis (ed. P. Fonagy), pp. 21-55. London: Psychoanalysis Unit, University College.
Hammersley, M. (1992) Deconstructing the qualitative-quantitative divide. In Mixing Methods: Qualitative and Quantitative Research (ed. J. Brannen), pp. 39-65. Aldershot: Avebury.
Harrington, R., Wood, A. & Verduyn, C. (1998) Clinically depressed adolescents. In Cognitive Behaviour Therapy for Children and Families (ed. P. J. Graham), pp. 156-193. Cambridge: Cambridge University Press.
Henggeler, S., Schoenwald, S.K., Borduin, C.M., et al (1998) Multisystemic Treatment of Antisocial Behavior in Children and Adolescents. New York: Guilford Press.
Jones, D. & Ramchandi, P. (1999) Child Sexual Abuse: Informing Practice from Research. Abingdon: Radcliffe Medical.
Kazdin, A. (1996) Dropping out of treatment. Clinical Child Psychology and Psychiatry, 1, 133-156.
Kazdin, A. (1997) Psychosocial treatments for conduct disorder. Journal of Child Psychology and Psychiatry, 38, 161-178.[Medline]
Kazdin, A., Siegel, T. & Bass, D. (1992) Cognitive problem solving skills and parent management training in the treatment of antisocial behaviors in children. Journal of Consulting and Clinical Psychology, 60, 733-747.[CrossRef][Medline]
Kendall, P. C. & Southam-Gerow, M. A. (1996) Longterm follow-up of a cognitive-behavioral therapy for anxiety-disordered youths. Journal of Consulting and Clinical Psychology, 64, 724-730.[CrossRef][Medline]
Klein, R. G., Abikoff, H., Klass, E., et al (1997) Clinical efficacy of methylphenidate in conduct disorder with and without attention deficit hyperactivity disorder. Archives of General Psychiatry, 54, 1073-1080.[Abstract]
Lask, B. & Matthew, D. (1979) Childhood asthma - a controlled trial of family psychotherapy. Archives of Diseases in Childhood, 54, 116-119.[Abstract]
McClellan, J. & Werry, J. (1994) Practice parameters for the assessment and treatment of children and adolescents with schizophrenia. Journal of the American Academy of Child and Adolescent Psychiatry, 33, 616-635.[Medline]
March, J. (1995) Cognitive behavioral therapy for children and adolescents with OCD: a review and recommendations for treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 34, 7-18.[Medline]
Mays, N. & Pope, C. (1996) Rigour and qualitative research. In Qualitative Research in Health Care (eds N. Mays & C. Pope), pp. 10-19. London: BMJ Publishing.
Medawar, P. (1984) Pluto's Republic, p. 97. Oxford: Oxford University Press.
Moran, G., Fonagy, P., Kurtz, A., et al (1991) A controlled study of the psychoanalytic treatment of brittle diabetes. Journal of the American Academy of Child and Adolescent Psychiatry, 30, 926-935.[Medline]
Muir Gray, J. A. (1997) Evidence-Based Health Care, p. 61. Edinburgh: Churchill-Livingstone.
Royal College of Psychiatrists' Research Unit (1999) FOCUS on the Use of Stimulants in Children with Attention-Deficit Hyperactivity Disorder: Primary Evidence-Based Briefing. London: Royal College of Psychiatrists' Research Unit.
Russell, G. F. M., Szmukler, G. I., Dare, C., et al (1987) An evaluation of family therapy in anorexia nervosa and bulimia nervosa. Archives of General Psychiatry, 44, 1047-1056.[Abstract]
Rutter, M., Maughan, B., Mortimore, P., et al (1979) Fifteen Thousand Hours. London: Open Books.
Sackett, D., Richardson, W. S., Rosenberg, W., et al (1997) Evidence-Based Medicine. Edinburgh: Churchill-Livingstone.
Sanders, M. R., Shepherd, R. W., Cleghorn, G., et al (1994) The treatment of abdominal pain in children: a controlled comparison of cognitive-behavioral family intervention and standard pediatric care. Journal of Consulting and Clinical Psychology, 62, 306-314.[CrossRef][Medline]
Shapiro, E. S., Shapiro, A. K., Fulop, G., et al (1989) Controlled study of haloperidol, pimozide and placebo for the treatment of Gilles de la Tourette Syndrome. Archives of General Psychiatry, 46, 722-730.[Abstract]
Skuse, D., Bentovim, A., Hodges, J., et al
(1998) Risk factors for development of sexually abusive
behaviour in sexually victimised adolescent boys: a cross-sectional study.
British Medical Journal,
317,
175-179.
Smith, P., Perrin, S. & Yule, W. (1998) Post-traumatic stress disorders. In Cognitive Behaviour Therapy for Children and Families (ed. P. J. Graham), pp. 127-142. Cambridge: Cambridge University Press.
Smith, R. (1999) The NHS: possibilities for the
endgame. British Medical Journal,
318, 210.
Strauss, A. & Corbin, J. (1998) Basics of Qualitative Research. London: Sage.
Szatmari, P. (1998) Evidence-based child
psychiatry and the two solitudes. Evidence-based Mental
Health, 2,
6-7.
Target, M. & Fonagy, P. (1996) The psychological treatment of child and adolescent psychiatric disorders. In What Works for Whom? (eds A. Roth & P. Fonagy), pp. 263-320. New York: Guilford Press.
Weiss, G. (1996) Attention deficit hyperactivity disorder. In Child and Adolescent Psychiatry (ed. M. Lewis), pp. 544-565. Baltimore, MD: Williams & Wilkins.
Weisz, J. R., Donenberg, G. R., Weiss, B., et al (1995) Bridging the gap between laboratory and clinic in child and adolescent psychiatry. Journal of Consulting and Clinical Psychology, 63, 688-701.[CrossRef][Medline]
Wiener, A., Withers, K., Patrick, M., et al
(1999) What changes are of value in severely disturbed
children? Clinical Child Psychology and Psychiatry,
4, 201-214.
Wilson, G. T., Eldredge, K. L., Smith, D., et al (1991) Cognitive behavioral treatment of bulimia nervosa: a controlled evaluation. Behaviour Research and Therapy, 29, 579-583.
Received for publication July 13, 1999. Revision received December 14, 1999. Accepted for publication December 16, 1999.