1 Epidemiology Unit, Centre for Research in Cancer Control, Queensland Cancer Fund, Brisbane, Queensland, Australia.
2 School of Population Health, University of Queensland, Queensland, Australia.
3 School of Public Health, Queensland University of Technology, Brisbane, Queensland, Australia.
4 National Cancer Control Initiative, Carlton, Victoria, Australia.
5 Health Information Centre, Queensland Health, Brisbane, Queensland, Australia.
6 Department of Community and Behavioral Health, College of Public Health, University of Iowa, Iowa City, IA.
Received for publication October 22, 2003; accepted for publication April 14, 2004.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
cross-over studies; data collection; mass screening; melanoma; randomized controlled trials; skin
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
A number of studies have compared the quality of data obtained through these two survey methods within independent samples (2, 1114); however, few studies have compared responses from the same participants using different survey methods. For example, OToole et al. (5) compared information on medical history, chemical exposure, and military history obtained by interviews and self-administered mailed questionnaires within a single cohort and found no significant differences in the test-retest reliability of answers. Brogger et al. (6) reported 79 percent agreement or higher for information on lung disease obtained from telephone interviews and mailed questionnaires.
As part of a large, randomized, community-based trial of population screening for melanoma (15), cross-sectional surveys of the prevalence of skin examination by a physician (clinical skin examination) and skin self-examination were conducted at annual intervals using telephone interviews and mailed questionnaires. Prior to analysis of these data, we wished to assess the comparability of information collected through these two methods and to determine the best method for future surveys, taking into account cost and data quality.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Of the 639 subjects randomized to receive the telephone interview first, all but two (99.7 percent) completed the interview. Of these, 564 (88.5 percent) returned the mailed questionnaire, giving us an overall completion rate for both the interview and the questionnaire of 88.3 percent. Of the 796 participants randomized to receive the mailed questionnaire first, 737 (92.6 percent) returned their questionnaire, and 706 (95.8 percent) of those persons completed their telephone interview 1 month later, giving us an overall completion rate for this group of 88.8 percent. The final sample for analysis comprised the 1,270 subjects (88.5 percent of the initial 1,435) who completed both the telephone interview and the mailed questionnaire.
Data collection
Interviews were conducted using computer-assisted telephone interview software. Participants randomized to receive the telephone interview first were asked for their name and address at the completion of the interview. Questionnaires were mailed 1 month later, followed after 2 weeks by a reminder letter and telephone calls to nonresponders. Participants randomized to receive the questionnaire first were asked for their names and addresses during the initial recruitment telephone call, and questionnaires were mailed within 3 days, followed after 2 weeks by reminder letters and telephone calls. In this group, telephone interviews were conducted 1 month after return of the questionnaire.
Skin examination by a physician and skin self-examination
Question wording was identical in the interview and the mailed questionnaire. Subjects were asked about whole-body skin examinations performed by a physician in the past 12 months and the past 3 years and about any skin examinations done by a physician in the past 12 months. Similarly, subjects were asked about whole-body skin self-examination or examination performed by a spouse or other nonmedical person in the past 12 months and about any skin self-examination or examination by a nonmedical person in the past 12 months. The exact wording of the questions is given in table 1.
|
Data management and analysis
Telephone interviews and mailed questionnaires were matched according to a unique identification number allocated to each participant. Information from mailed questionnaires was double-entered and verified using the SAS statistical software program PROC COMPARE (SAS, version 8; SAS Institute, Inc., Cary, North Carolina).
Comparison of telephone and postal surveys
For each of the questions about skin examination listed above, the extent of agreement between interviews and mailed questionnaires was assessed as the percentage of subjects who gave the same response for both methods. Unweighted kappa statistics were used to measure agreement while correcting for chance (16). Kappa scores between 0.81 and 1.00 were regarded as indicating "almost perfect" agreement, scores between 0.61 and 0.80 as indicating "substantial" agreement, and scores between 0.41 and 0.60 as indicating "moderate" agreement (17) (table 1). Kappa scores of 0.60 or higher and/or overall agreement of at least 80 percent were considered necessary to conclude that results obtained by interview and by mailed questionnaire were similar enough to be interchangeable. Confidence intervals around kappa scores were calculated according to the formula of Fleiss (16). Univariate and multivariate logistic regression was used to examine associations between percentage of agreement and sociodemographic factors, attitudes towards skin cancer, and skin cancer risk factors. The frequencies of missing and "dont know" responses were compared between interviews and mailed questionnaires. All analyses were undertaken using SAS, version 8.
Survey costs
The costs of using telephone interviews were calculated by adding the interviewer salaries, telephone costs, and staff costs in programming the interview. The costs of using postal surveys included printing costs and postal charges (including reply-paid envelopes), the costs of remailing surveys and making follow-up telephone calls to nonresponders, and data entry.
Ethics
Ethical approval for this study was granted by the Behavioural and Social Sciences Ethical Review Committee of the University of Queensland.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Agreement between telephone interviews and mailed questionnaires
Overall agreement between the two survey methods for the two questions about whole-body clinical skin examinations was 91.2 percent (examinations in the last 12 months) and 88.6 percent (examinations in the last 3 years), respectively (table 1). Kappa scores were 0.69 and 0.70, reflecting substantial agreement. Agreement was somewhat lower (82.4 percent; = 0.64) for clinical skin examinations of "any" part of the body in the last 12 months. Irrespective of whether the interview or the questionnaire was administered first, the estimated prevalences of clinical skin examinations in the past 12 months or 3 years were similar for telephone interviews and mailed questionnaires.
With regard to whole-body skin self-examinations in the past 12 months, 81.9 percent of subjects gave the same response in their interview and their questionnaire ( = 0.46). Agreement was lowest when participants were asked about skin self-examination of "any" part of the body in the last 12 months (70.2 percent agreement;
= 0.41). In comparison with mailed questionnaires, telephone interviews resulted in an apparently higher estimated prevalence of whole-body skin self-examination in the past 12 months (23.1 percent vs. 18.8 percent; p = 0.03) and a higher estimated prevalence of "any" skin self-examination (55.4 percent vs. 46.9 percent; p = 0.02) (table 1). The direction of this difference was consistent regardless of the order of administration of the survey method, although the magnitude of the difference was more pronounced for the group that received the telephone interview before the mailed questionnaire.
Factors associated with agreement between the two survey methods
With regard to whole-body clinical skin examination in the past 12 months, agreement between the telephone interview and the mailed questionnaire was slightly higher for women and for subjects with a higher education, although these findings did not reach statistical significance (table 2). Agreement was significantly lower for subjects who had had a mole or spot removed from their skin in the past. This was the only variable associated with agreement in the multivariate logistic regression analysis (odds ratio = 0.49, 95 percent confidence interval: 0.31, 0.79). There was no association between overall agreement and employment status, tendency to burn if exposed to the sun for more than half an hour without protection, concern about a mole, or order of randomization (telephone or postal survey first). With regard to whole-body skin self-examination, agreement was higher for women, subjects with less than a high school education, and persons with a darker hair color (table 2). Agreement was significantly lower for subjects who thought they were very likely to develop skin cancer in the future. This was also the only variable associated with agreement in the multivariate logistic regression analysis (odds ratio = 0.57, 95 percent confidence interval: 0.35, 0.92).
|
Costs of data collection using the two survey methods
The cost of collecting data using mailed questionnaires, including the costs of formatting, printing, mailing and remailing, making reminder telephone calls, and data entry, was A$3.01 (US$1.96) per completed questionnaire. The cost of collecting data using the telephone interview, including the costs of computer-assisted interview programming, interviewer salaries, and telephone charges, was more than triple that of the mailed questionnaire at A$9.55 (US$6.21) per completed interview.
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Estimated prevalences of clinical skin examination reported here are similar to those observed in an earlier telephone survey conducted within the population from which the current sample was drawn (18, 20) and similar for the two methods within this study. In contrast, there was a tendency for subjects in the present study to report a history of skin self-examination more often in telephone interviews than in mailed questionnaires, particularly in response to the question about "any" skin self-examination.
Surprisingly, agreement between telephone interviews and mailed questionnaires was significantly lower in the multivariate analysis for participants reporting a history of having a spot or mole removed. For skin self-examination, those who believed they were likely to develop skin cancer in the future received lower agreement scores in the multivariate analysis than other participants. These participants might examine their skin more frequently but may not always recall this behavior as a skin self-examination.
Although there were few missing data for either method, the mailed questionnaires yielded relatively more missing data than the telephone interviews, similar to the case in earlier reports (5, 6, 14, 19). The costs of the telephone interviews were considerably higher than those for mailed questionnaires, a fact also reported in other contexts (5, 14). Within the present study, following initial recruitment, response rates between the two survey modes were comparably high. For other mailed questionnaires conducted within the context of the randomized trial of melanoma screening, response rates of close to 75 percent have been achieved, although other investigators have reported lower rates of response to mailed questionnaires as compared with telephone surveys (2123). Asch et al. (24) examined 321 mailed surveys and found a mean response rate of 60 percent.
Provided that the same question wording is used, the results of surveys of clinical skin examinations conducted by mailed questionnaire or telephone interview are likely to be directly comparable. In situations where response rates from mailed questionnaires are lower than those for telephone interviews, the higher costs of the telephone interview may be justified. Our results suggest that telephone surveys of skin self-examination produce higher prevalence estimates than surveys using mailed questionnaires, though this is less pronounced when investigators use well-defined questions about whole-body skin self-examination than when they ask about "any" skin self-examination. This finding should be considered in light of the substantial additional costs associated with telephone interviews. Overall, the similarity of results obtained by telephone interview and mailed questionnaire, coupled with the much higher cost of the telephone interviews, suggests that, provided that response rates are similar, use of self-administered mailed questionnaires is an appropriate method for conducting surveys of the prevalence of skin screening by physicians.
![]() |
ACKNOWLEDGMENTS |
---|
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|