1 Health Risk Reduction Projects, Department of Psychiatry, University of California, Los Angeles, Los Angeles, CA.
2 Westat, Inc., Gaithersburg, MD.
3 Department of Geographic Medicine, University of Alabama at Birmingham, Birmingham, AL.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
adolescence; computers; evaluation studies; HIV; marijuana abuse
Abbreviations: ACASI, audio computer-assisted self-interview; EMIT, enzyme-multiplied immunoassay technique; HIV, human immunodeficiency virus; REACH, Reaching for Excellence in Adolescent Care and Health.
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
While some studies report high rates of agreement between self-reports and objective measures (57
), others have found extremely low agreement rates, with some of the lowest rates reported among studies that have included younger populations (3
). In a review of 13 studies published up to 1985 (6
), agreement rates were only moderate. A more recent meta-analytical review of 24 studies published since 1985 (3
), which included only studies that used a biological criterion of validity (e.g., urinalysis, hair analysis), reported that the median conditional kappa (kc) was 0.42. This is notably below the level that typically represents an acceptable level of report accuracy. Of the studies reviewed, only five were conducted with juveniles or young adults. The kc means for each of these five studies ranged from -0.05 to 0.79.
Subject characteristics and experimental study conditions both appear to affect reliability and accuracy of self-reports of drug use (2). Early studies found inaccurate reports of drug use associated with severity of arrest charges (8
) and with the number of prior arrests (9
). Magura et al. (6
) found that higher criminality was associated with underreporting, and in their meta-analytic review, Magura and Kang (3
) also found that criminal justice populations, especially youthful offenders, report among the lowest validity coefficients. In addition to subject characteristics, differing methodologies may be partially responsible for variability in findings across studies. The method of data collection can affect survey measurements of sensitive behaviors (10
); privacy of response may be especially important for information about illegal drug use activities. Studies comparing interviewer administration with self-administration have found that self-administration increases the reporting of abortions (11
), alcohol consumption (12
), and illicit drug use (13
). Because of such findings, respondents are often asked to complete paper-and-pencil, self-administered questionnaires; however, self-administered questionnaires require respondents to be sufficiently literate as well as to be able to follow the flow of an interview, such as branching and skip patterns. A new technology, audio computer assisted self interview (ACASI), has recently been used to administer complex, sensitive questions. In an ACASI assessment format, respondents hear, over headphones, spoken questions that have been recorded and stored on a computer and see them displayed on screen. Response choices are heard through headphones, as well as lighted on screen. Advantages are that respondents may answer sensitive questions privately, that even respondents with limited reading ability can respond, that data may be encrypted immediately after completion of the survey by the respondent to ensure confidentiality, that the flow of the questionnaire (i.e., looping and branching) is done automatically by the computer system, and that the administration is standardized. Particularly for adolescents, the computer administration format may seem more interesting than traditional survey methodology. In a comparison of three methods of data collectioncomputer-assisted personal interview, computer-assisted self interview, and ACASIwhile response rate did not differ by mode of data collection, self-administration increased the proportion of adult respondents who admitted that they had used illicit drugs in some time frames (14
). Specifically, ACASI yielded the highest percentage of reported marijuana for lifetime use, and although no significant differences were found for the previous year or previous month, means were in the expected direction. In the 1995 National Survey of Adolescent Males (15
), 1,690 respondents were randomly assigned to answer questions through ACASI or through a traditional self-administered questionnaire. Large differences were found for very sensitive behaviors: Estimates of prevalence for male-male sex, injection drug use, and sexual contact with intravenous drug users were higher by factors of three or more when ACASI was used. Additionally, higher reporting of crack and cocaine use was found, as well as higher reporting for some other risk behaviors.
In this study, validity of self-report of marijuana use among HIV-infected adolescents and HIV-seronegative adolescents at risk for HIV was investigated. Self-report of marijuana use was obtained through ACASI, and urine drug testing for cannabinoids was completed at a centralized certified laboratory.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Data for this analysis were from the 6-month visits of subjects in the REACH cohort, since questions about marijuana use were not included in the baseline questionnaire. At the time of analysis, the REACH cohort consisted of 376 adolescent subjects, of whom 276 had completed a 6-month visit. Of those 276 subjects, 195 had completed both a urinalysis and self-report of drug use as part of their 6-month study visit. Of these, 13 had nonmatching dates for the urinalysis and interview and were therefore excluded from the analysis, leaving 182 subjects whose data were analyzed. Of these 182 subjects, 67 percent are HIV positive, and 73 percent are female. The racial/ethnic breakdown of the sample is as follows: 66 percent Black/African American; 22 percent mixed race or other; and 12 percent White. The mean age of the sample was 17.5 years (standard deviation = 1.2; range, 1320).
Procedures
Informed consent.
Some institutional review boards required parental permission for participation of youth in this study, while others did not; requirements were followed at each site. Informed consent was obtained from all adolescents participating in the study.
Interview.
Data used in this study were obtained from four sources: direct, face-to-face interview; ACASI; urine drug screening; and chart review. The face-to-face interview is conducted by the study coordinator, most often a nurse or nurse-practitioner who has been trained in both interviewing techniques and the specific assessment. Items in the face-to-face interview have a specific question-by-question narrative to facilitate accurate and reliable data collection. For the ACASI, questions on the computer screen are read to the subject through earphones to assist individuals with low reading levels. Responses are highlighted on the screen as they are read aloud to the subject to minimize error response. The ACASI utilizes time line reminders throughout to promote recall for specified time frames, as well as help screens. The ACASI was specifically designed for the project and was previewed by the REACH community advisory board; items were pilot tested by using pen and paper at five sites during the development phase. When the ACASI assessment is completed by a participant, data are encrypted automatically and transferred to the data center. Chart review was completed by the study coordinator.
Assessment
Subject characteristics.
Age, race/ethnicity, and education level were assessed in the face-to-face interview. In addition, history of juvenile delinquency was assessed by asking the adolescents, "Were you ever held in a detention center or jail for at least 2 nights in a row?" and "In your whole lifetime, how long have you spent in a detention center or jail?" They were also asked whether they had ever supported themselves by trading sex for money or drugs, selling drugs, gambling (e.g., running numbers, etc.), stealing (e.g., food, stereos, vans, cars, etc.), or panhandling or scamming. CD4+ T-cell counts and quantitative HIV-1 viral load testing were completed by using standardized tests as previously described (16).
ACASI.
To assess self-reported marijuana use, subjects were initially asked whether they had used marijuana in the previous 3 months, and no further questions were asked if they responded negatively. If they answered affirmatively, they were queried about frequency and asked, "Did you smoke marijuana today or yesterday?" If they again responded affirmatively to this question, no further questions were asked. If they denied use of marijuana "today or yesterday," they were asked: "How many days ago did last smoke marijuana?"
Urinalysis.
Drug screening for the REACH protocol was conducted on urine samples collected every 3 months. Urines were tested for marijuana at a certified commercial laboratory by using the enzyme-multiplied immunoassay technique (EMIT) with a detection cutoff of 100 ng/ml. All positives were confirmed by gas chromatography. Urine samples were checked for adulteration and quality assurance by using measurements of pH, urine-specific gravity, and urine creatinine measurements.
At the detection of 100 ng/ml, any use of marijuana in the previous 24 hours should be detected (17). Detection beyond 24 hours depends on the amount of usage, the potency of inhaled or ingested marijuana, and general smoking/inhalation behaviors (18
). Given the long terminal half-life in plasma (2535 hours), it may be possible to detect cannabinoids in the urine as many as 10 days postusage (18
, 19
). While there is evidence of individual variation in kinetics, there are no reported kinetic differences between men and women (18
, 19
).
For analyses, use of marijuana was defined in four ways. The first was by a positive urinalysis for cannabinoids. For a detection period of 48 hours, the urinalysis may have some false-positive results (i.e., the detected use occurred more than 48 hours previously), but for a detection period of more than 48 hours, the urinalysis may have false-negative results because of the loss of sensitivity. Therefore, three self-report definitions of marijuana use were established from the ACASI: use today or yesterday, use within the last 5 days, and use within the last 7 days. The first self-report definition corresponds approximately to the 48-hour period for which the EMIT test should be sensitive, and the last definition corresponds to the approximate maximum length of time for which the EMIT test could be sensitive. The 5-day definition was established so that we could determine whether there was a trend in increasing agreement between self-report and urinalysis as the time period was increased.
Statistical methods.
Concordance between self-reports and urinary marijuana assays was assessed by the Cohen's kappa statistic (20), which measures the degree of agreement between two classificatory variables. It provides a measure of agreement between two items, adjusting for any agreement that would occur by chance, treating the sources symmetrically. Perfect agreement is indicated by k = 1, and chance agreement only by k = 0. (Negative k indicates less than chance agreement.) Typically, a kappa of greater than 0.75 is considered to indicate good agreement; 0.40 to 0.75, fair agreement; and less than 0.40, poor agreement. Kappa is most appropriate as a measure of agreement when the time periods covered by self-report and the criterion (in this case, the urinalysis) are similar. For questions with discrepant time periods, a more interpretable coefficient of agreement is conditional kappa (kc), which also measures the extent to which agreement of two sources exceeds chance; however, kc assumes one source (here, the urinalysis) to be a "gold standard" and measures the other source's agreement with that standard. Persons with a positive test who fail to report use are considered to be inaccurate reporters, but self-reports of drug use by subjects with negative biological tests are ignored in the computation of kc so that the coefficient does not reflect inconsistencies based on the results of honest disclosure of use during periods in which the test may be unable to detect actual use. Kappa and conditional kappa were computed for each stratum such as male gender or positive HIV status. Pooled estimates, summarizing the stratified values, were obtained by the method of Barlow et al. (21
), and their homogeneity (e.g., equality of kappa or conditional kappa for males and females) were tested by Cochran's Q statistic. Guidelines for acceptable values of kappa are those in Landis and Koch (22
), and these were also applied to conditional kappa. Hypothesis tests for the indices were based on comparison of a standardized estimate to a normal deviate. Multiple comparison corrections were not applied, and two-sided p values of 0.05 or less were considered significant.
Sensitivity calculates the proportion of those with a positive urine test for any substance (true positives) who actually report use of the substance (self-reported positives divided by urine test positives). Cross-tabulations are used to examine bivariate relations between substance use underreporting and subjects characteristics (i.e., race/ethnicity, gender, and criminal behavior or detention). The "percent agreement" is equivalent to a sensitivity calculation if urinalysis is considered to be the truth. However, a specificity calculation was not performed because of problems with the sensitivity of urinalysis that were described above.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
Comparison of urinalysis results with reported marijuana use by ACASI
As described above, the prevalence rates of marijuana use by self-report for all three time periods were generally higher than the detection rate by urinalysis. Further reflecting the imperfect concordance of self-report and urinalysis, the presence of illegal behaviors was significantly associated with ACASI self-reports but not with the rate of positive urinalysis. For the 2-day self-report period, the overall summary kc is 0.57, which is considered fair. The summary kcs are 0.71 and 0.69 for the 5- and 7-day reporting periods, respectively, which represent moderate agreement. The improvement in kc values from the 2-day period to the longer periods most likely reflects the sensitivity of the urine screen to marijuana use more than 2 days earlier. Similarly, the percent of self-reports that agreed with positive urinalyses was only 64 percent for the 2-day period, but increased to 79 percent for the 5- and 7-day periods.
Differences in reporting between HIV-positive and HIV-negative subjects
The seven possible covariates of marijuana use listed above were also examined as possible mediators of the relation of self-report to urinalysis. For each factor and self-report time period (within 2, 5, or 7 days), kc was computed to assess whether agreement rates between positive urinalyses and positive self-reports were greater than would be found by chance. Tables 2 and 3 display the kc values for each stratum (e.g., HIV positive and HIV negative, male or female, and a summary weighted average of the kc values). For the demographic and behavioral variables that were examined, significant differences among the kc values, suggesting variation in self-reports about true marijuana use, are found only for categories of HIV status. When there is objectively confirmed marijuana use (i.e., a positive urine test), those who are HIV positive are much more likely to give a positive self-report with a 5- or 7-day reporting period than are those who are HIV negative. This is seen both for unadjusted percent agreement values (over 90 percent for HIV positive vs. 50 percent for HIV negative) and for the chance-adjusted kc values, which are over 0.9 for the HIV-positive subjects and range from 0.3 to 0.36 for the HIV-negative subjects. These two kc values are considered good and poor, respectively, and differ significantly from one another by Cochran's Q statistic (p = 0.005). None of the other subject features, such as education, illegal behaviors, or having been held in detention, were associated with substantial or statistically significant differences in the concordance of self-reports and positive urinalyses.
|
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
When performed under chain-of-custody procedures in a certified laboratory, urinalysis is appropriate for drug treatment and enforcement purposes, since a positive cannabinoid result is almost always truly positive for recent marijuana use. However, it may be less useful for epidemiologic and behavioral studies in which prevalence estimates are to be made. As described in Materials and Methods, the underlying assumption of the conditional kappa calculation is that positive self-reports are true self-reports that reflect the insensitivity of the biological assay for the time period under study. Although this assumption might not be entirely correct, it is reasonable to assume that most positive self-reports collected in the manner utilized in this study (e.g., data encrypted and not available to clinicians who work with the adolescent; sent directly to the research management center) are true.
The percent of positive urinalyses and of self-reported marijuana use for each of our three time periods is seen in table 1. For all three periods, more subjects reported marijuana use through ACASI than were identified by urinalysis. The prevalence estimates are about the same for the 2-day period, but the prevalence estimates for 5 and 7 days are substantially higher by self-report than by urinalysis. In fact, for the 7-day period, the prevalence rate by self-report is more than double that by urinalysis. Surprisingly, although urinalysis should be highly sensitive for marijuana use in the previous 48 hours, our study had 12 subjects who had negative urinalyses but who reported using marijuana during that period. Some of these discrepancies might be explained by recall problems (i.e., some subjects who may have telescoped earlier use into this reporting period), but this could also indicate that the assay is somewhat less sensitive than was believed, even during the 48-hour period, and perhaps especially for very low-dose usage. It is also possible that these subjects simply exaggerated their marijuana use and that the ACASI reports were actually false positives, while the urinalyses were accurate. However, given the pattern of findings over the 2-, 5-, and 7-day periods and the method of data collection (i.e., subjects could not "impress" anyone; subjects were aware that items were computer administered and that nurses and physicians at the site never had access to the encrypted data, which was sent directly to the research coordinating center), this seems less likely.
For purposes of estimating the true prevalence of use of marijuana in a population, our data suggest that maximum sensitivity may be achieved through combining ACASI and urine analysis, although if only one modality is feasible, ACASI may be superior for prevalence of use measurement or estimation. Some users will not admit true marijuana use, as is confirmed by those in our study who had positive urinalyses but did not admit use even in the previous 7 days. However, if the false-negative percentage of self-reports could be reliably estimated, an adjustment to the self-report rate can be made to estimate more accurately the true prevalence of use. Since there is currently no true gold standard assay that is both highly sensitive and highly specific for marijuana use within specific time periods, such as 2 or 7 days, an estimate of the rate of false-negative self-reports would need to rely on statistical inference. In our study, the 2-day self-reports were negative for 10 of 28 subjects (36 percent) who were urinalysis positive, but the 7-day self-reports were negative for only six of these 28 (21 percent). Urinalysis almost certainly identifies some marijuana use longer than 2 days in the past. Therefore, 36 percent might be considered the point estimate of the upper bound of the false-negative rate for self-reports. Conversely, nearly all positive urinalyses should indicate marijuana use in the previous 7 days, but some use in that period might not be detected by urinalysis. Therefore, 21 percent might be considered the point estimate of the lower bound. A range of prevalence estimates based only on self-reports could be made by using these point estimates (or their confidence intervals) of the upper and lower bounds of the false-negative self-report rates. In this study, there are wide confidence intervals, given the small number of subjects. However, if larger studies were available, upper and lower bounds could be estimated more precisely.
In addition to the issue of what the gold standard is to which self-report is compared, the validity of self-report of drug use may be influenced by what specific drug is targeted for report. Although marijuana was the only drug able to be assessed for validity of self-report in this sample because of low prevalence of use of other drugs, in other studies, self-reports for some substances have been found to be more valid than others. Brown et al. (5) found that among admissions to an inpatient substance abuse treatment unit, self-reports of recent alcohol and cocaine use were more valid than were those for marijuana. The authors postulated several explanations for that finding: marijuana use may be more detrimental to memory function than alcohol or cocaine are; typical marijuana users may consider admission of marijuana to be more socially deviant; or perhaps more likely, marijuana users may perceive their marijuana use in a very casual manner and thus have decreased their focus on its use. However, the opposite finding has also been reported: juvenile arrestees are less reluctant to indicate use of marijuana than of a less acceptable substance, such as cocaine (4
). Additionally, of the five studies conducted with juveniles or young adults in the meta-analysis by Magura and Kang (3
), the kc means were as follows: -0.05 (cocaine only, single time frame); 0.34 (multiple drugs; multiple time frames; range, 0.040.76); 0.35 (multiple drugs; multiple time frames; range, 0.180.64); 0.00 (cocaine only, single time frame); and 0.79 (marijuana only, two time frames). As can be seen, the reporting of marijuana showed the highest validity across those studies.
Finally, subject characteristics may influence validity of report. Interestingly, for the 5-day reporting period, 94 percent of the HIV-positive subjects versus 50 percent of HIV-negative subjects who had positive urinalyses acknowledged smoking marijuana. For an illegal behavior such as this, the level of self-report among HIV-positive adolescents is exceptionally high. HIV-positive subjects are struggling with numerous issues, including fluctuations in their health, side effects of medications that may be impacted by other drug use, and other psychosocial issues. It may be that they have become accustomed to relying on their medical care providers and to disclosing personal information. It may also be that they have been stigmatized due to their HIV and that the additional label of marijuana user is not perceived as negative to them relative to their serostatus. In addition, in this sample, adolescents who had been in held in detention had higher marijuana use rates than did those who had not, so that having been held in detention was a significant predictor of positive self-reports of use as well as of a positive urinalysis. This is consistent with other studies; for example, more than 60 percent of the adolescents held in detention facilities test positive for marijuana (23), whereas only approximately 18 percent of nonincarcerated adolescents acknowledged using marijuana in the 1997 Monitoring the Future Study (24
). However, the validity of self-report for those in this study who had been held in detention was not significantly lower than for those who had not been held in detention. Although this would seem to contradict previous findings, it may be that juvenile offenders are less likely to have a valid self-report only when they are currently in detention or on probation and, thus, when consequences could be more severe for admitting to illegal drug use.
Several limitations to this study should be noted. The subjects knew that urinalysis for illegal substances was planned. However, they also knew that the clinic staff would not be apprised of the results. It is not clear whether self-reports would be as accurate if no objective biological assessment were planned. Therefore, it may still be necessary to combine biological measures (or the possibility of biological measures) with self-reports to assure reasonable accuracy. This study also examined only one method of data collection for self-reportsACASI. Other studies have shown that ACASI may increase self-report of sensitive behaviors, but further research is warranted in adolescents to compare the utility of various data collection methods for marijuana and for other drugs.
Maximum sensitivity was obtained from a combination of ACASI and urine drug testing. Contrary to previous studies, our data suggest that if a single evaluative instrument is to be used for general prevalence of marijuana use rather than exact time of usage, ACASI would be more sensitive than urine drug testing for marijuana among adolescents overall and particularly among HIV-infected adolescents.
![]() |
ACKNOWLEDGMENTS |
---|
The following investigators, listed in order of the numbers of subjects enrolled, are participating in this study: University of Miami: Dr. L. Friedman, L. Pall, D. Maturo, and A. Pasquale; Montefiore Medical Center: Dr. D. Futterman, D. Monte, M. Alovera-DeBellis, N. Hoffman, and S. Jackson; Children's Hospital of Philadelphia: Dr. B. Rudy, M. Tanney, Dr. D. Schwarz, and A. Feldman; Children's Hospital of Los Angeles: Dr. M. Belzer, D. Tucker, C. Kallal, and D. Fuchs; Tulane Medical Center: Dr. S. E. Abdalian, L. Green, M. Ales, and L. Wenthold; Children's National Medical Center: Dr. L. J. D'Angelo, C. Trexler, C. Townsend-Akpan, R. Hagler, and J. A. Morrissy; University of Maryland: Dr. L. Peralta, C. Ryder, and S. Miller; Cook County Hospital/University of Chicago: Dr. L. Henry-Reid, R. Camacho, and Dr. D. Johnson; Children's Hospital, Birmingham: Dr. M. Sturdevant, A. Howell, and J. E. Johnson; Children's Diagnostic and Treatment Center: Dr. A. Puga, D. Cruz, and P. McLendon; Emory University: Dr. M. Sawyer and J. Tigner; St. Jude Children's Research Hospital: Dr. P. Flynn, K. Lett, and J. Doss; Mt. Sinai Medical Center: Dr. L. Levin and M. Geiger; University of Medicine and Dentistry of New Jersey: Dr. P. Stanford and F. Briggs; State University of New York Health Science Center at Brooklyn: Dr. J. Birnbaum and Dr. M. Ramnarine.
The following investigators have been responsible for the basic science agenda: Center for Virology, Immunology, and Infectious Disease, Children's Research Institute, Children's National Medical Center, Dr. C. Holland; University of California at San Francisco, Dr. A. B. Moscicki; University of California at Los Angeles, Dr. D. A. Murphy; University of Alabama at Birmingham, Dr. S. H. Vermund; The Fearing Laboratory, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, Dr. P. Crowley-Nowick; University of Pennsylvania and the Children's Hospital of Philadelphia, Dr. S. D. Douglas. Network operations and analytic support are provided by Dr. C. M. Wilson at the C. Partlow at University of Alabama and B. Hobbs, Dr. J. H. Ellenberg, L. Paolinelli, S. J. Durako, Dr. L. Muenz, R. Mitchell, K. Clingan, P. Ohan, V. Junankar, O. Leytush, A. Bennett, M. Rakheja, C. Xue, Y. Ma, and J. Houser at Westat, Inc.
Staff from sponsoring agencies include Dr. A. Rogers, (National Institute of Child Health and Human Development), K. Davenny, Dr. V. Smeriglio (National Institute of Drug Abuse), E. Matzen (National Institute of Allergy and Infectious Diseases), Dr. B. Vitiello (National Institutes of Mental Health), and G. Weissman (Health Resources and Services Administration).
The investigators are grateful to the members of the Community Advisory Board for their insight and counsel.
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|