Comparability of Skin Screening Histories Obtained by Telephone Interviews and Mailed Questionnaires: A Randomized Crossover Study

Joanne F. Aitken1,2 , Philippa H. Youl1, Monika Janda1,3, Mark Elwood4, Ian T. Ring5 and John B. Lowe6

1 Epidemiology Unit, Centre for Research in Cancer Control, Queensland Cancer Fund, Brisbane, Queensland, Australia.
2 School of Population Health, University of Queensland, Queensland, Australia.
3 School of Public Health, Queensland University of Technology, Brisbane, Queensland, Australia.
4 National Cancer Control Initiative, Carlton, Victoria, Australia.
5 Health Information Centre, Queensland Health, Brisbane, Queensland, Australia.
6 Department of Community and Behavioral Health, College of Public Health, University of Iowa, Iowa City, IA.

Received for publication October 22, 2003; accepted for publication April 14, 2004.


    ABSTRACT
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
The comparability of information collected through telephone interviews and information collected through mailed questionnaires has not been well studied. As part of the first phase of a randomized controlled trial of population screening for melanoma in Queensland, Australia, the authors compared histories of skin examination reported in telephone interviews and self-administered mailed questionnaires. A total of 1,270 subjects each completed a telephone interview and a mailed questionnaire 1 month apart in 1999; 564 subjects received the interview first, and 706 received the mailed questionnaire first. Agreement between the two methods was 91.2% and 88.6% for whole-body skin examination by a physician in the last 12 months and the last 3 years, respectively, and 81.9% for whole-body skin self-examination in the last 12 months. Agreement was lower for "any" skin self-examination. Agreement between the two methods was similar regardless of whether the interview or the questionnaire was administered first. Missing data were less frequent for interviews (0.5%) than for mailed questionnaires (3.8%). Costs were estimated at A$9.55 (US$6.21) per completed interview and A$3.01 (US$1.96) per questionnaire. The similarity of results obtained using telephone interviews and mailed questionnaires, coupled with the substantially higher cost of telephone interviews, suggests that self-administered mailed questionnaires are an appropriate method of assessing this health behavior.

cross-over studies; data collection; mass screening; melanoma; randomized controlled trials; skin


    INTRODUCTION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Personal or telephone interviews are commonly used to collect information on health behaviors (14). While they are generally viewed as providing good-quality information with few missing data (5, 6), interviews are relatively expensive to conduct and are not always suited to sensitive topics or questions that require personal reflection, such as psychological scales. A desire for respondents to provide socially acceptable answers and interviewer and other biases associated with telephone interviews are well recognized (2, 79). Self-administered mailed questionnaires are relatively inexpensive to use (2, 5, 10). However, respondents must be able to read; the interpretation of questions and therefore the accuracy of responses can vary widely, depending on question wording and format; and incomplete or missing responses are likely to be more frequent (5, 10, 11).

A number of studies have compared the quality of data obtained through these two survey methods within independent samples (2, 1114); however, few studies have compared responses from the same participants using different survey methods. For example, O’Toole et al. (5) compared information on medical history, chemical exposure, and military history obtained by interviews and self-administered mailed questionnaires within a single cohort and found no significant differences in the test-retest reliability of answers. Brogger et al. (6) reported 79 percent agreement or higher for information on lung disease obtained from telephone interviews and mailed questionnaires.

As part of a large, randomized, community-based trial of population screening for melanoma (15), cross-sectional surveys of the prevalence of skin examination by a physician (clinical skin examination) and skin self-examination were conducted at annual intervals using telephone interviews and mailed questionnaires. Prior to analysis of these data, we wished to assess the comparability of information collected through these two methods and to determine the best method for future surveys, taking into account cost and data quality.


    MATERIALS AND METHODS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Study design and subjects
A randomized crossover design was used to assess the comparability of information on history of skin examination reported in telephone interviews and mailed questionnaires. The study was conducted as part of the first phase of a randomized community-based trial of population screening for melanoma in Queensland, Australia (15). Subjects were sampled at random from one community in the intervention group and one community in the control group using a commercially available telephone list (Desktop Marketing Systems, Melbourne, Australia) after exclusion of all telephone numbers selected for a recent survey of skin screening behavior. Eligible subjects were 1) resident in one of the two communities; 2) aged 30 years or more; 3) able to understand English; and 4) available during the study period. Sampling was stratified by gender. Of the 6,364 telephone numbers called, contact was made with 1,964 eligible subjects, of whom 1,435 (73.1 percent) agreed to take part. Of these persons, 639 were randomized using a computer program to receive the telephone interview first, followed 1 month later by a mailed questionnaire. The remaining 796 subjects received the mailed questionnaire first, followed 1 month later by the telephone interview. We oversampled those receiving the mailed questionnaire first in (incorrect) anticipation of lower response rates in this group.

Of the 639 subjects randomized to receive the telephone interview first, all but two (99.7 percent) completed the interview. Of these, 564 (88.5 percent) returned the mailed questionnaire, giving us an overall completion rate for both the interview and the questionnaire of 88.3 percent. Of the 796 participants randomized to receive the mailed questionnaire first, 737 (92.6 percent) returned their questionnaire, and 706 (95.8 percent) of those persons completed their telephone interview 1 month later, giving us an overall completion rate for this group of 88.8 percent. The final sample for analysis comprised the 1,270 subjects (88.5 percent of the initial 1,435) who completed both the telephone interview and the mailed questionnaire.

Data collection
Interviews were conducted using computer-assisted telephone interview software. Participants randomized to receive the telephone interview first were asked for their name and address at the completion of the interview. Questionnaires were mailed 1 month later, followed after 2 weeks by a reminder letter and telephone calls to nonresponders. Participants randomized to receive the questionnaire first were asked for their names and addresses during the initial recruitment telephone call, and questionnaires were mailed within 3 days, followed after 2 weeks by reminder letters and telephone calls. In this group, telephone interviews were conducted 1 month after return of the questionnaire.

Skin examination by a physician and skin self-examination
Question wording was identical in the interview and the mailed questionnaire. Subjects were asked about whole-body skin examinations performed by a physician in the past 12 months and the past 3 years and about any skin examinations done by a physician in the past 12 months. Similarly, subjects were asked about whole-body skin self-examination or examination performed by a spouse or other nonmedical person in the past 12 months and about any skin self-examination or examination by a nonmedical person in the past 12 months. The exact wording of the questions is given in table 1.


View this table:
[in this window]
[in a new window]
 
TABLE 1. Agreement between telephone interviews and mailed questionnaires for questions relating to clinical skin examination and skin self-examination (n = 1,270), Queensland, Australia, 1999*
 
Additional information collected during the telephone interview
During the telephone interview, subjects were asked about their level of education, their employment status, their perceived likelihood of getting skin cancer in the future, whether they had a spot on their skin that they were concerned about, skin cancer risk factors (hair color, tendency to sunburn after unprotected sun exposure), and whether they had ever had a mole or other spot removed from their skin.

Data management and analysis
Telephone interviews and mailed questionnaires were matched according to a unique identification number allocated to each participant. Information from mailed questionnaires was double-entered and verified using the SAS statistical software program PROC COMPARE (SAS, version 8; SAS Institute, Inc., Cary, North Carolina).

Comparison of telephone and postal surveys
For each of the questions about skin examination listed above, the extent of agreement between interviews and mailed questionnaires was assessed as the percentage of subjects who gave the same response for both methods. Unweighted kappa statistics were used to measure agreement while correcting for chance (16). Kappa scores between 0.81 and 1.00 were regarded as indicating "almost perfect" agreement, scores between 0.61 and 0.80 as indicating "substantial" agreement, and scores between 0.41 and 0.60 as indicating "moderate" agreement (17) (table 1). Kappa scores of 0.60 or higher and/or overall agreement of at least 80 percent were considered necessary to conclude that results obtained by interview and by mailed questionnaire were similar enough to be interchangeable. Confidence intervals around kappa scores were calculated according to the formula of Fleiss (16). Univariate and multivariate logistic regression was used to examine associations between percentage of agreement and sociodemographic factors, attitudes towards skin cancer, and skin cancer risk factors. The frequencies of missing and "don’t know" responses were compared between interviews and mailed questionnaires. All analyses were undertaken using SAS, version 8.

Survey costs
The costs of using telephone interviews were calculated by adding the interviewer salaries, telephone costs, and staff costs in programming the interview. The costs of using postal surveys included printing costs and postal charges (including reply-paid envelopes), the costs of remailing surveys and making follow-up telephone calls to nonresponders, and data entry.

Ethics
Ethical approval for this study was granted by the Behavioural and Social Sciences Ethical Review Committee of the University of Queensland.


    RESULTS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Characteristics of participants
The sociodemographic characteristics and skin cancer risk profile of participants were similar to those reported previously for this population (18): approximately 50 percent were ≥50 years of age, 41.2 percent were employed full-time, 47.6 percent had at least a high school education, 73.4 percent stated that their skin burned after unprotected exposure to the sun, 22.0 percent were currently concerned about a mole or spot on their skin, and 54.7 percent had had a mole or skin spot removed in the past. The two groups randomized to receive either the telephone interview or the mailed questionnaire first were similar in terms of all measured characteristics, except that those who received the mailed questionnaire first were more likely to be unemployed (43.9 percent vs. 40.4 percent), less likely to have at least a high school education (44.1 percent vs. 52.0 percent), and less likely to believe that they would get skin cancer in the future (28.0 percent vs. 36.2 percent).

Agreement between telephone interviews and mailed questionnaires
Overall agreement between the two survey methods for the two questions about whole-body clinical skin examinations was 91.2 percent (examinations in the last 12 months) and 88.6 percent (examinations in the last 3 years), respectively (table 1). Kappa scores were 0.69 and 0.70, reflecting substantial agreement. Agreement was somewhat lower (82.4 percent; {kappa} = 0.64) for clinical skin examinations of "any" part of the body in the last 12 months. Irrespective of whether the interview or the questionnaire was administered first, the estimated prevalences of clinical skin examinations in the past 12 months or 3 years were similar for telephone interviews and mailed questionnaires.

With regard to whole-body skin self-examinations in the past 12 months, 81.9 percent of subjects gave the same response in their interview and their questionnaire ({kappa} = 0.46). Agreement was lowest when participants were asked about skin self-examination of "any" part of the body in the last 12 months (70.2 percent agreement; {kappa} = 0.41). In comparison with mailed questionnaires, telephone interviews resulted in an apparently higher estimated prevalence of whole-body skin self-examination in the past 12 months (23.1 percent vs. 18.8 percent; p = 0.03) and a higher estimated prevalence of "any" skin self-examination (55.4 percent vs. 46.9 percent; p = 0.02) (table 1). The direction of this difference was consistent regardless of the order of administration of the survey method, although the magnitude of the difference was more pronounced for the group that received the telephone interview before the mailed questionnaire.

Factors associated with agreement between the two survey methods
With regard to whole-body clinical skin examination in the past 12 months, agreement between the telephone interview and the mailed questionnaire was slightly higher for women and for subjects with a higher education, although these findings did not reach statistical significance (table 2). Agreement was significantly lower for subjects who had had a mole or spot removed from their skin in the past. This was the only variable associated with agreement in the multivariate logistic regression analysis (odds ratio = 0.49, 95 percent confidence interval: 0.31, 0.79). There was no association between overall agreement and employment status, tendency to burn if exposed to the sun for more than half an hour without protection, concern about a mole, or order of randomization (telephone or postal survey first). With regard to whole-body skin self-examination, agreement was higher for women, subjects with less than a high school education, and persons with a darker hair color (table 2). Agreement was significantly lower for subjects who thought they were very likely to develop skin cancer in the future. This was also the only variable associated with agreement in the multivariate logistic regression analysis (odds ratio = 0.57, 95 percent confidence interval: 0.35, 0.92).


View this table:
[in this window]
[in a new window]
 
TABLE 2. Factors associated in univariate analysis with agreement between telephone interviews and mailed questionnaires for whole-body skin examination by a doctor and whole-body self-examination in the past 12 months (n = 1,270), Queensland, Australia, 1999
 
Extent of incomplete data from the two survey methods
The frequency of item omission (defined as missing or "don’t know" responses) was significantly lower for the telephone interview (0.5 percent) than for the self-administered mailed questionnaire (3.8 percent) (p < 0.001). The mean number of item omissions for the telephone interview was 0.08 (range, 0–2), while the corresponding number for the mailed questionnaire was 0.5 (range, 0–5), with 2.4 percent of participants having failed to provide a "yes/no" answer for two or more questions. Item omission occurred more frequently among older respondents, those who were less well educated, and those who were currently not working.

Costs of data collection using the two survey methods
The cost of collecting data using mailed questionnaires, including the costs of formatting, printing, mailing and remailing, making reminder telephone calls, and data entry, was A$3.01 (US$1.96) per completed questionnaire. The cost of collecting data using the telephone interview, including the costs of computer-assisted interview programming, interviewer salaries, and telephone charges, was more than triple that of the mailed questionnaire at A$9.55 (US$6.21) per completed interview.


    DISCUSSION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
To our knowledge, this is the first study to have compared two standard survey methods—telephone interviews and mailed questionnaires—for assessing the prevalence of skin cancer screening. We found agreement between the two methods of more than 90 percent for reports of whole-body skin examinations by a physician in the past 12 months and 89 percent for the past 3 years, and somewhat lower agreement of more than 80 percent for reports of "any" skin examinations by a physician in the past 12 months. There was also overall agreement of more than 80 percent for whole-body skin self-examination in the past 12 months, while agreement was 70 percent for "any" skin self-examination in the past 12 months. The kappa values for both self-examination measures obtained through the two survey methods point towards moderate agreement. Our results are within the range reported previously for comparisons between telephone interviews and mailed questionnaires in the assessment of other health outcomes, specifically respiratory symptoms (82–100 percent agreement) (6) and musculoskeletal conditions (70–88 percent) (19).

Estimated prevalences of clinical skin examination reported here are similar to those observed in an earlier telephone survey conducted within the population from which the current sample was drawn (18, 20) and similar for the two methods within this study. In contrast, there was a tendency for subjects in the present study to report a history of skin self-examination more often in telephone interviews than in mailed questionnaires, particularly in response to the question about "any" skin self-examination.

Surprisingly, agreement between telephone interviews and mailed questionnaires was significantly lower in the multivariate analysis for participants reporting a history of having a spot or mole removed. For skin self-examination, those who believed they were likely to develop skin cancer in the future received lower agreement scores in the multivariate analysis than other participants. These participants might examine their skin more frequently but may not always recall this behavior as a skin self-examination.

Although there were few missing data for either method, the mailed questionnaires yielded relatively more missing data than the telephone interviews, similar to the case in earlier reports (5, 6, 14, 19). The costs of the telephone interviews were considerably higher than those for mailed questionnaires, a fact also reported in other contexts (5, 14). Within the present study, following initial recruitment, response rates between the two survey modes were comparably high. For other mailed questionnaires conducted within the context of the randomized trial of melanoma screening, response rates of close to 75 percent have been achieved, although other investigators have reported lower rates of response to mailed questionnaires as compared with telephone surveys (2123). Asch et al. (24) examined 321 mailed surveys and found a mean response rate of 60 percent.

Provided that the same question wording is used, the results of surveys of clinical skin examinations conducted by mailed questionnaire or telephone interview are likely to be directly comparable. In situations where response rates from mailed questionnaires are lower than those for telephone interviews, the higher costs of the telephone interview may be justified. Our results suggest that telephone surveys of skin self-examination produce higher prevalence estimates than surveys using mailed questionnaires, though this is less pronounced when investigators use well-defined questions about whole-body skin self-examination than when they ask about "any" skin self-examination. This finding should be considered in light of the substantial additional costs associated with telephone interviews. Overall, the similarity of results obtained by telephone interview and mailed questionnaire, coupled with the much higher cost of the telephone interviews, suggests that, provided that response rates are similar, use of self-administered mailed questionnaires is an appropriate method for conducting surveys of the prevalence of skin screening by physicians.


    ACKNOWLEDGMENTS
 
This study was supported by the Queensland Cancer Fund and Queensland Health.


    NOTES
 
Correspondence to Dr. J. F. Aitken, Queensland Cancer Fund, P.O. Box 201, Spring Hill, Queensland 4004, Australia (e-mail: joannea{at}qcfepi.org.au). Back


    REFERENCES
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Donovan RJ, Homan CD, Corti B, et al. Face-to-face household interviews versus telephone interviews for health surveys. Aust N Z J Public Health 1997;21:134–40.[ISI][Medline]
  2. Rhodes T, Girman, CJ, Jacobsen SJ, et al. Does the mode of questionnaire administration affect the reporting of urinary symptoms? Urology 1995;46:341–5.[CrossRef][ISI][Medline]
  3. Norman SA, Localio AR, Zhou L, et al. Validation of self-reported screening mammography histories among women with and without breast cancer. Am J Epidemiol 2003;158:264–71.[Abstract/Free Full Text]
  4. Chapman RS, Hadden WC, Perlin SA. Influences of asthma and household environment on lung function in children and adolescents: The Third National Health and Nutrition Examination Survey. Am J Epidemiol 2003;158:175–89.[Abstract/Free Full Text]
  5. O’Toole BI, Battistutta D, Long A, et al. A comparison of costs and data quality of three health survey methods: mail, telephone and personal home interview. Am J Epidemiol 1986;124:317–28.[Abstract]
  6. Brogger J, Bakke P, Eide GE, et al. Comparison of telephone and postal survey modes on respiratory symptoms and risk factors. Am J Epidemiol 2002;155:572–6.[Abstract/Free Full Text]
  7. Lyberg L, Kasprzyk D. Data collection methods and measurement error: an overview. In: Biemer PP, Groves RM, Lyberg LE, et al, eds. Measurement errors in surveys. New York, NY: John Wiley and Sons, 1991:237–57.
  8. Dillman DA, Tarnai J. Mode effects of cognitively designed recall questions: a comparison of answers to telephone and mail surveys. In: Biemer PP, Groves RM, Lyberg LE, et al, eds. Measurement errors in surveys. New York, NY: John Wiley and Sons, 1991:74–93.
  9. Abrahamson JH. Survey methods in community medicine. Edinburgh, United Kingdom: Churchill Livingstone, 1974.
  10. Bradburn NM, Sudman S. The current status of questionnaire design. In: Biemer PP, Groves RM, Lyberg LE, et al, eds. Measurement errors in surveys. New York, NY: John Wiley and Sons, 1991:29–40.
  11. Kleinman L, Leidy NK, Crawley J, et al. A comparison trial of paper-and-pencil versus computer administration of the Quality of Life in Reflux and Dyspepsia (QOLRAD) questionnaire. Med Care 2001;39:181–9.[CrossRef][ISI][Medline]
  12. Walker AH, Restuccia JD. Obtaining information on patient satisfaction with hospital care: mail versus telephone. Health Serv Res 1994;19:291–306.
  13. Brambilla DJ, McKinlay SM. A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. Am J Epidemiol 1987;126:962–71.[Abstract]
  14. McHorney CA, Kosinski M, Ware JE Jr. Comparisons of the costs and quality of norms for the SF-36 health survey collected by mail versus telephone interview: results from a national survey. Med Care 1994;32:551–67.[ISI][Medline]
  15. Aitken JF, Elwood JM, Lowe JB, et al. A randomised trial of population screening for melanoma. J Med Screen 2002;9: 33–7.[CrossRef][ISI][Medline]
  16. Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York, NY: John Wiley and Sons, 1981.
  17. Maclure M, Willett WC. Misinterpretation and misuse of the kappa statistic. Am J Epidemiol 1987;126:161–9.[ISI][Medline]
  18. Janda M, Elwood M, Ring IT, et al. Prevalence of skin screening by general practitioners in regional Queensland. Med J Aust 2004;180:10–15.[ISI][Medline]
  19. Ooijen M, Ivens U, Johansen C, et al. Comparison of a self-administered questionnaire and a telephone interview of 146 Danish waste collectors. Am J Ind Med 1997;31:653–8.[CrossRef][ISI][Medline]
  20. Aitken JF, Janda M, Lowe JB, et al. Prevalence of whole-body skin self-examination in a population at high risk for skin cancer (Australia). Cancer Causes Control 2004;15:453–63.[CrossRef][ISI][Medline]
  21. Perkins JJ, Sanson-Fisher RW. An examination of self- and telephone-administered modes of administration for the Australian SF-36. J Clin Epidemiol 1998;51:969–73.[CrossRef][ISI][Medline]
  22. Galobardes B, Sunyer J, Anto JM, et al. Effect of method of administration, mail or telephone, on the validity and reliability of the respiratory health questionnaire. The Spanish Centres of the European Asthma Study. J Clin Epidemiol 1998;51:875–81.[CrossRef][ISI][Medline]
  23. Fowler FJ, Gallagher PM, Stringfellow VL, et al. Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Med Care 2002;40:190–200.[CrossRef][ISI][Medline]
  24. Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129–36.[CrossRef][ISI][Medline]




This Article
Abstract
FREE Full Text (PDF)
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Disclaimer
Request Permissions
Google Scholar
Articles by Aitken, J. F.
Articles by Lowe, J. B.
PubMed
PubMed Citation
Articles by Aitken, J. F.
Articles by Lowe, J. B.