1 Department of Psychological Medicine, University of Glasgow, Gartnavel Royal Hospital, 2 Centre for Applied Social Psychology, Department of Psychology and 3 Department of Psychology, University of Strathclyde and 4 Department of Psychology, Glasgow Caledonian University, Glasgow, UK
(Received 5 September 2002; first review notified 29 October 2002; in revised form 5 June 2003; accepted 30 June 2003)
* Author to whom correspondence should be addressed at present address: Department of Clinical Psychology, Royal Hospital for Sick Children, Yorkhill NHS Trust, Dalnair Street, Glasgow G3 8SJ. Tel: 0141 201 0644; Fax: 0141 201 9246; E-mail: suzyharper{at}hotmail.com
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Fundamental to this issue is the way verbal reports are conceptualized. They may be viewed as a direct window into the mind, by way of cognitions. However, they could also be viewed as discursive acts that are behaviours in their own right, having function and meaning only within a given context (Wittgenstein, 1963; Potter and Wetherell, 1987
; Potter and Edwards, 1992
). In practice, this distinction leads to a polarity in research methods, notably the preference for a quantitative rather than a qualitative approach. This paper presents a methodology that allows a reconciliation between the requirements of hard science and the later Wittgenstein-type philosophy of language as primarily performative in context.
The principles of Signal Detection Theory (SDT) (Shannon and Weaver, 1949; Green and Swets, 1966
; Corso, 1970
) may be borrowed, by analogy, to effect such a reconciliation. In SDT, there is no assumption of a correlation of 1.0 between a stimulus event and a verbal report. Specifically, the same event is not expected to lead to the same verbal report. This variability in verbal reports may be due to internal factors like fatigue or attention but it can also be affected by externally manipulable factors like motivation or schedules of reward. In a typical SDT study, rewards or disincentives are offered for different responses and these alter the individual's motivation to verbally respond in one way rather than another.
Although the constituent parts of a social-scientific study are less easily, and less precisely, manipulated than they can be in a SDT/psychophysical study, there is no reason to suppose that the SDT view of language as being open to motivated bias would not apply in a social as compared to a psychophysical context. Davies and Best (1996) have termed this approach Social Criterion Theory (SCT) and argue that it provides an integrative model of the processes that mediate self-reported information.
In SCT, three concepts are borrowed from SDT and applied by analogy as follows. (1) In a SDT trial signal strength is represented by the intensity of the physical signal in conjunction with the observer's perceptual acuity. SCT argues that signal strength is analogous to the method used to elicit self-reported information, for example, a questionnaire would have a higher signal strength that an open-ended interview because it provides more cues for the respondent regarding the expected response. (2) Response bias represents the individual's tendency to say yes a signal was presented. This measure is an additive function of all trials where the participant says that a signal was presented regardless of whether or not they were correct in this statement. In SDT, response bias is the factor that is affected by schedules of reward or punishment and changing individual motivation. The SCT analogy uses this concept to represent how individuals decide their response in a social situation. Therefore, response bias is a function of the demand characteristics of the interviewing situation and the individuals' interpretation of these in light of their own past experiences and their motivation to respond in one manner over another depending on their desired outcome. (3) Within SDT, sensitivity yields a measure of the acuity of the individual's sensory system interacting with the signal strength. In SCT, sensitivity is used to represent the individual's understanding of the issue in conjunction with the amount of information about the issue available in the environment.
In the present study, the utility of SCT was tested using a methodology devised to mirror the conditions of a SDT experiment but using meaningful verbal stimuli. These stimuli comprised 40 custom written vignettes, which described individuals drinking alcohol and systematically incorporated the diagnostic criteria for both alcohol dependence and alcohol intoxication that are operationally defined within DSM-IV (American Psychiatric Association, 1994). Alcohol dependence criteria were used to represent signal trials and alcohol intoxication criteria were used to represent noise trials.
Alcohol treatment programmes frequently report attrition rates of around 50%, yet individuals who are accepted for treatment have usually verbalized their commitment to the programme. Thus, in such alcohol treatment programmes, many participants provide verbal reports that conflict with their subsequent behaviour. A previous study (Harper, 1999) used a Social Criterion methodology to demonstrate differences in response bias between heavy drinkers who had presented for treatment compared to heavy drinkers who had not. Those who had presented for treatment were more likely to categorize vignettes as problem alcohol use (had a riskier response bias) than those who had not presented for treatment. However, there were no differences between the two groups in terms of sensitivity. It was then argued that these differences in response bias reflected the differences in motivation between the two groups with regard to their continued alcohol use. Although both groups could discriminate between problematic and less problematic alcohol use (i.e. their sensitivity levels were equivalent), the group who had presented for treatment were more likely to verbally report problematic alcohol use reflected by their relatively risky response bias scores. This demonstrated that those individuals who were more willing to identify other people's alcohol consumption as problematic were more likely to have sought assistance with their own alcohol use in the form of an alcohol treatment programme.
The present study applied the Social Criterion methodology prospectively to ascertain whether differences in response bias could discriminate between those individuals who presented for treatment for problem alcohol use and completed treatment compared with those who presented for treatment and subsequently dropped out. Any differences in response bias could form the basis of a predictive pre-treatment screening tool that did not rely on the truth or falsehood of participants' responses but looked instead at their motivation to provide one verbal response over another in a clinical context. Two hypotheses were considered, as follows. (1) Patients who drop-out of a treatment programme for problem alcohol use would have more conservative response bias scores (i.e. would be more reluctant to classify vignettes as problem alcohol use) than those who remained in treatment for a 4 week minimum period and those individuals who have been in treatment beyond the 4 week minimum period would have riskier response bias scores (i.e. would be more likely to classify vignettes as problem alcohol use) than either of the other two groups of patients. (2) All three groups would have equivalent and high sensitivity scores because they would all be knowledgeable about the characteristics of alcohol misuse.
![]() |
SUBJECTS, MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
Development of the vignette methodology
Forty vignettes were written, which described either problem or normal alcohol consumption. (Copies of the 40 vignettes are available from the first author.) DSM-IV diagnostic criteria for substance abuse (F10.1) and substance dependence (F10.2) (American Psychiatric Association, 1994) were used to represent problem alcohol use by incorporating them into the vignettes systematically. In terms of the SDT analogy these represented signals. Vignettes describing normal social drinking were constructed around DSM-IV diagnostic criteria for alcohol intoxication (F10.00). In SDT terms these represented noise. The diagnostic criteria employed in DSM-IV are considered to have equal weighting so it was assumed that the vignettes would have equal signal strength as they each contained a description of one diagnostic criterion. Support for this assumption is available from the authors of DSM-IV who acknowledge that flexibility is necessary in reaching a diagnosis using their diagnostic criteria and warn against a cookbook approach. However, they also state that an overly flexible approach to diagnostic criteria reduces the utility of DSM-IV as a common means of communication about mental disorders (DSM-IV, 1994, p. xxiii). This would seem to indicate that the diagnostic criteria are regarded, for practical purposes, as having equivalent weighting. Furthermore, it is implicit in the instructions given that the diagnostic criteria have equal weightings since diagnosis may be reached by ticking a set number of criteria from the list provided. It does not matter which specific criteria are ticked. It is the aggregate score that is important. Therefore, they are all given equal value in diagnostic terms.
All diagnostic criteria were represented with the exception of those pertaining to withdrawal and tolerance (F10.2_1 and F10.2_2). Pilot testing indicated that these represented higher signal strength (represented a more problematic level of problem alcohol use) in comparison with the other criteria. DSM-IV stipulates that any diagnosis of substance dependence should specify whether physiological dependence is present or not (DSM-IV, 1994, p. 185). This suggests that substance dependence with physiological dependence represents a more severe type of disorder than non-physiological substance dependence. If the criteria for withdrawal and tolerance had been included in vignettes they would have been immediately identifiable as problem alcohol use. SDT requires ambiguity between signal and noise stimuli and participants showed no uncertainty during the pilot study regarding these two diagnostic criteria.
All vignettes were written in straightforward language and were as entertaining and as brief as possible. Pilot testing suggested that short and entertaining vignettes provided an aid to comprehension and helped to maintain participants' interest in the task. Vignettes were controlled for the gender and occupational status of the protagonist. Men and women were represented equally and all vignettes stated that the protagonist was employed, as unemployment has been shown to correlate with alcohol related problems (Allsop and Saunders, 1984).
A further pilot study examined the relative signal strengths of the signal and noise vignettes. Nineteen individuals who had presented with problem alcohol use (mean age ± SD: 46.79 ± 7.15; male:female ratio: 15:4), all outpatients at a hospital alcohol unit, were asked to dichotomously rate the 40 vignettes (20 signal and 20 noise) according to whether they thought the vignette represented problem or non-problem alcohol use. A related t-test was performed on the raters' total scores which indicated that mean scores were significantly different [t(19) = -4.96, P = 0.001]. Concurrent with signal detection protocols, signal and noise distributions should overlap to an extent (i.e. there must be some ambiguity or participants would respond correctly every time). The mean sensitivity score of the 19 participants was 0.99 with a standard deviation of 0.85. This indicated that the signal and noise distributions were about one standard deviation apart which is an acceptable level of ambiguity for a signal detection experiment (MacMillan and Creelman, 1991). Two examples of vignettes, one signal and one noise, are shown below together with the DSM-1V code and description that they represent.
Example of signal vignette
F10.2_(7): The substance use is continued despite knowledge of having a persistent or recurrent physical or psychological problem that is likely to have been caused or exacerbated by the substance (e.g. current cocaine use despite recognition of cocaine-induced depression, or continued drinking despite recognition that an ulcer was made worse by alcohol consumption) (Amercian Psychiatric Association, 1994, p. 187).
Simon was a middle-aged journalist, who worked on a busy National newspaper. Like most journalists, Simon enjoyed a drink, frequently going to the pub both at lunchtime and after work. Recently, Simon's Doctor had diagnosed a stomach ulcer and had advised Simon to cut down on alcohol and spicy foods. However, Simon did not feel able to stop going to the pub because that was where he got a lot of information about new stories. He had cut down substantially on spicy food though, and he figured that should help a bit with the ulcer.
Example of noise vignette
F10.00: (A) Recent ingestion of alcohol. (B) Clinically significant maladaptive behavioural or psychological changes (e.g. inappropriate sexual or aggressive behaviour, mood lability, impaired judgment, impaired social or occupational functioning) that developed during or shortly after, alcohol ingestion. (C) One (or more) of the following signs, developing during, or shortly after, alcohol use: (1) slurred speech, (2) incoordination, (3) unsteady gait, (4) nystagmus, (5) impairment in attention or memory, (6) stupor or coma. (D) The symptoms are not due to another medical condition and are not better accounted for by another mental disorder. (Amercian Psychiatric Association, 1994, p. 201).
It was Saturday night and Dorothy was drunk. It was her friend's birthday and a group of them were out celebrating by going on a pub-crawl. They had started drinking at 5 o-clock and it was now half past 10. They had only managed to visit three pubs in this time and they were determined to fit in at least three more before they went clubbing. Shabout time we left this pub, slurred Dorothy to her friends before tripping on her handbag and falling over.
Materials
Forty vignettes were employed: 20 described problem alcohol use (signals) and 20 depicted nonproblem alcohol use (noise). A four-point confidence rating scale was employed, as follows: (1) I am very sure this is problem alcohol use; (2) I am fairly sure this is problem alcohol use; (3) I am fairly sure this is not problem alcohol use; and (4) I am very sure this is not problem alcohol use.
Procedures
All those patients who agreed to participate were breathalysed by hospital staff (as part of their existing treatment protocol) to ensure sobriety. Individuals who had zero breath alcohol levels were subsequently admitted to the study. An information sheet that briefly explained the purpose of the study was given to all participants and then the researcher explained to the groups (which ranged in size from two to 12 people) that she was interested in individual's perceptions of other people's alcohol use. She also explained that she was particularly interested in their opinions because they had experienced personal problems with alcohol use. Participants were first asked to read and answer questions on a trial vignette. This was used as a measure of baseline reading comprehension. The trial vignette was similar to those used in the actual study except it had five short answer questions accompanying it. Those participants who did not answer all five questions correctly were excluded from the analyses (four individuals from the long-term group and two individuals from the drop-out group failed to complete the reading comprehension test satisfactorily). Next, participants were asked to read each of the 40 experimental vignettes, which were presented randomly, and rate on the confidence scales the extent to which they believed that each vignette described problem or nonproblem alcohol use.
Analyses
Traditional SDT experiments usually involve many more trials than it was possible to present using the above methodology. To ascertain that the mathematical principles of SDT analyses were not violated by this relative paucity of trials it was necessary to generate Z-transformed receiver operator characteristic (ROC) curves that should be straight lines (this indicates that the signal and noise distributions of responses are normal) and have unit slope (this suggests that the standard deviation of the signal and noise distributions are equal). To plot useful ROC curves it is necessary to generate at least three levels of response bias from each participant and in the present study confidence rating scales were used to this end. Ratings at the 1 level (I am very sure this is problem alcohol use) were divided by 20 (the number of presentations of signal and noise trials) for both the signal and noise ratings, then ratings at the 1 and 2 (I am fairly sure this is problem alcohol use) levels were aggregated and divided by 20 for both signal and noise trials, then ratings at the 1, 2 and 3 (I am fairly sure this is not problem alcohol use) were aggregated and divided by 20 for both signal and noise trials. This yielded three measures of response bias for each participant (one measure from each probability of hit and probability of false alarm pair). Z-transformed ROC curves were generated by transforming the three mean probabilities of hit and probabilities of false alarm pairs for each group (see: MacMillan and Creelman, 1991 for further explanation of this procedure). These indicated that the data were appropriate for SDT analyses (see Table 2).
|
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
There were no statistically significant differences between response bias scores for the successful completion group and the long-term group. However, there was a trend towards the long-term group having riskier response bias scores (see Fig. 1) and it is possible that a larger sample size would have demonstrated this effect to a statistically significant level. Further research is required to clarify this issue. If it were the case that response bias scores became progressively more risky with increased commitment to treatment (reflecting increased motivation to maintain abstinence) then response bias scores could be used at the start of a treatment programme to predict likelihood of success and those individuals at risk of dropping out of treatment could be targeted for extra support.
Sensitivity scores did not differ, and were high, across all three groups. SCT argues that sensitivity reflects the individual's knowledge about the topic combined with the amount of information available about the topic in the environment. Therefore, it was understandable that sensitivity would be equivalent in this sample because they were all heavy drinkers and were, arguably, very familiar with the symptoms of problem alcohol use. It is possible that sensitivity would be more discriminatory in populations who were not such heavy drinkers. In addition, we would argue that alcohol use and the problems associated with it are familiar issues within western society. Therefore, it is likely that most individuals who live in a culture where alcohol use is common would have reasonably high sensitivity scores for this issue. However, sensitivity may well be an important factor in decisions about how to respond regarding less familiar, but similarly sensitive, issues (e.g. drug use or eating disorders).
The methodology described above mimics SDT protocols but presented fewer trials than would be employed in a traditional SDT experiment, where the stimuli may be dim lights or faint sounds. Therefore, it is possible that the data generated in the present study may have been less suitable for SDT analyses. The latter require two main assumptions to be met: the signal and noise distributions should be normally distributed and their standard deviations should be equal. These properties of the data are reflected in the Z-transformed ROC curve which should have unit slope (showing that the standard deviations of the signal and noise distributions of responses were equal) and should be straight lines (showing that the signal and noise distributions formed normal distributions). Table 2 shows the regression equations for the Z-transformed ROC curves for the data for the three groups. These equations show that the data for the three groups are as acceptable as psychophysical data in meeting the requirements for SDT analyses (MacMillan and Creelman, 1991).
These findings are interesting because they predict attrition when it was not possible to achieve this via traditional methods e.g. in terms of individual's stated commitment to the treatment programme or through their clinical/demographic presentation (see Table 1). There is an extensive literature that examines the utility of the verbal information provided by individuals who experience difficulties with their alcohol use (Midanik, 1988; Babor et al., 1990
; McKay, 1999
) and many variables have been proposed that mediate the production of alcohol misusers' verbal responses. However, the overall focus of this literature is on obtaining a reliable or true account. We argue that this approach is misconceived and suggest that participants say they are committed to treatment because if they did not say this, their behaviour (obtaining a referral to an alcohol unit) would be incongruous with their verbalizations. So, in a clinical context, alcohol misusers will emphasize problems with alcohol use and they will state that they are committed to treatment. If this is correct, verbal information will never discriminate usefully between individuals who will drop-out of treatment and those who will successfully complete treatment if it is taken at face value and assumed to be simply true or false.
The SCT methodology described in this paper suggests a different epistemological basis for verbal reports. We argue that in sensitive contexts verbal reports do not directly reflect reality and we concur with Wittgenstein's philosophy of language. There is a requirement for new approaches to the collection and analysis of verbal data that are commensurate with this epistemological view and this paper has presented data to suggest that the SCT analogy can be useful to this end. Better prediction can only be obtained with a more sophisticated approach to verbal data and its analysis.
![]() |
Acknowledgements |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders, 4th edn. American Psychiatric Association, Washington DC.
Babor, T. F., Brown, J. and Del Boca, F. K. (1990) Validity of self-reports in applied research on addictive behaviours: Fact or fiction? Behavioural Assessment 12, 531.
Bandura, A. (1982) Self-Efficacy Mechanisms in Human Agency. American Psychologist 27, 122147.[CrossRef]
Corso, J. F. (1970) The Experimental Psychology of Sensory Behaviour. Holt, Rhineholt and Winston, London.
Davies, J. B. and Best, D. W. (1996) Demand characteristics and research into drug use. Psychology and Health 11, 291299.[ISI]
Green, D. M. and Swets, J. A. (1966) Signal Detection Theory and Psychophysics. John Wiley & Sons, New York.
Harper, S. M. (1999) An Alternative Methodology For Alcohol Misusers Self-Report. Alcohol Update 39, 35. The Scottish Council on Alcohol, Glasgow.
McKay, J. R. (1999) Studies of Factors in Relapse to Alcohol, Drug and Nicotine Use: A Critical Review of Methodologies and Findings. Journal of Studies on Alcohol 60(4), 566576.[ISI][Medline]
MacMillan, N. A. and Creelman, C. D. (1991) Detection Theory: A Users Guide. Cambridge University Press, Cambridge.
Midanik, L. T. (1988) Validity of self-reported alcohol use: A literature review and assessment. British Journal of Addiction 83, 10191029.[ISI][Medline]
Potter, J. and Edwards, D. (1992) Discursive Psychology. Sage, London.
Potter, J. and Wetherell, M. (1987) Discourse and Social Psychology: Beyond Attitudes and Behaviour. Sage, London.
Shannon, C. E. and Weaver, W. (1949) The Mathematical Theory of Communication. University of Illinois, Illinois.
Wittgenstein, L. (1963) Philosophical Investigations, 2nd edn. Oxford University Press, Oxford.