1 Cancer Prevention Program, Fred Hutchinson Cancer Research Center, Seattle, WA
2 Department of Epidemiology, School of Public Health and Community Medicine, University of Washington, Seattle, WA
3 Department of Community and Family Medicine, Dartmouth Medical School, Hanover, NH
Correspondence to Dr. Emily White, Cancer Prevention Program, Fred Hutchinson Cancer Research Center, 1100 Fairview Avenue North, P.O. Box 19024, Seattle, WA 98109 (e-mail: ewhite{at}fhcrc.org).
Received for publication December 16, 2004. Accepted for publication March 22, 2005.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
data collection; epidemiologic methods; questionnaires
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Achieving high response rates at reasonable costs can often be challenging. Numerous randomized trials of methods of increasing response rates to mailed questionnaires have been conducted (13
). In an excellent review and meta-analysis of 292 such trials, Edwards et al. (2
) identified several factors that increase response rates. Approaches that had the strongest effects (odds ratios of 1.42.4, or approximately a 1020 percentage point increase for response rates of 4060 percent) were (from the greatest effect to the least): a more salient survey topic; recorded (e.g., certified) delivery; including a monetary incentive; a shorter questionnaire versus a longer questionnaire; including incentives along with the questionnaire versus upon return of the questionnaire; preliminary contact with potential participants; follow-up of nonrespondents; and including the questionnaire with the follow-up contacts (2
).
Meta-analyses of trials of nonmonetary incentives (pencils, pens, and small gifts) show that they have a smaller positive effect on response ratesin the range of a 58 percentage point increase, on average (24
). The majority of the trials were of surveys in the social sciences or in business/marketing or surveys of health professionals, rather than population-based health surveys. To our knowledge, only three studies have tested whether including a pencil or pen with a population-based, mailed health questionnaire increases survey response, and none of them reported a significant increase (5
7
).
In this paper, we report on two randomized controlled trials in which a pen or pencil was randomly included or not included in a second mailing sent to nonresponders to the first mailing. In the first study, the survey was a recruitment mailing for a study on women's health. In the second, the survey was a follow-up mailing sent 2 years after men and women completed a baseline cohort study questionnaire. We determined how inclusion of a pen or pencil in the second mailing affected response rates and costs. We analyzed costs to understand whether the additional cost of the pen or pencil was compensated for by the reduced number of nonrespondents who would need to be contacted in a third round of mailings.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Women were identified as eligible for recruitment using a population-based mammography registry, the New Hampshire Mammography Network, through which permission was given to contact participants for research purposes. Further details about the design and development of the mammography registry are provided elsewhere (8). Recruitment for the NHWH Study involved sending a personalized cover letter about the study, a consent form, and a 12-page questionnaire, which women were asked to complete and return with their consent forms by mail. After an initial mailing with no study incentives was sent in July 2001 to 39,050 women, 3,550 were found to be ineligible (because of death, moving out of the area, or an undeliverable first mailing); this left 35,500 eligible women. Of those women, 24,814 responded; thus, 10,686 women were eligible for the second-mailing study (pen vs. no pen).
A pen with the study logo printed on it was included in the remail of the initial recruitment materials to randomly selected (with 50 percent probability) nonresponders (n = 5,364), while the remaining nonresponders (n = 5,322) were remailed study recruitment materials with no pen. We stratified our response rates on the basis of when the surveys were returned (within 30 days or within 3160 days). All study activities were approved by Dartmouth College's Committee for the Protection of Human Subjects.
Study 2The VITamins And Lifestyle Study
The primary aim of study 2 was to investigate the most effective strategies for minimizing cost and enhancing response rates to a follow-up questionnaire in the 77,700-person VITamins And Lifestyle (VITAL) Study cohort (9). Study 2 was conducted among Vanguard participants in the VITAL cohort. The Vanguard participants were 573 men and women in 13 counties of western Washington State who had completed the VITAL baseline questionnaire during a pilot phase of the VITAL Study in 2000.
In JuneDecember 2002, approximately 2 years after completion of the baseline questionnaire, we conducted study 2 among the Vanguard participants to assess response to a follow-up questionnaire using three mailings. We excluded 40 Vanguard participants who had recently been contacted for a pilot study for DNA collection, as well as four who had died, six who had moved out of the study area, and 71 who were out of the target age range of 5076 years; this left 452 participants for the mailing study. We determined after mailing that 13 others were ineligible because of death, moving out of the study area, or having an undeliverable questionnaire with no new address being found; this left 439 eligible men and women in the mailing study.
The first mailing contained a personalized cover letter, a questionnaire, and a postage-paid return envelope, which was followed 2 weeks later by a reminder postcard. A total of 298 subjects returned the questionnaire, leaving 141 nonrespondents for the randomized trial. The second questionnaire was sent to the nonrespondents 2 months after the initial mailing. Among those persons, we randomly selected a group (with 50 percent probability) to receive a study-logo pencil with the questionnaire packet. A third mailing was sent to nonresponders 2 months after the second mailing. The Fred Hutchinson Cancer Research Center's institutional review board approved this study and all related materials.
Statistical analyses
In study 1, we assessed differences in response rates between the pen and no-pen groups, using chi-squared tests. In study 2, we used Mantel-Haenszel methods for odds ratios and p values, to allow adjustment for sex, and converted the sex-adjusted odds ratio to sex-adjusted response rates for the pencil and no-pencil groups. For both studies, we computed an estimated response rate (RR) across the first two mailings as RRi = RRm1 + (1 RRm1) RRm2i, where i refers to the pen/pencil group versus the no-pen/-pencil group, m1 refers to mailing 1, and m2i refers to mailing 2 in group i.
We estimated how the observed differences in response rates between the groups would affect costs for a mailing protocol of up to three mailings per person, when the second mailing included a pencil. Estimated response rates for mailings 1 and 2 were based on the average from studies 1 and 2, and the estimated response rate to mailing 3 was based on study 2. Estimated costs were obtained from study 2 vendors and the US Postal Service in 2003.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
|
|
Table 4 shows projected response rates and costs of a mailing protocol consisting of three questionnaire mailings, comparing mailings that include a pencil in the second mailing only with those that do not. Response numbers and costs are based on 10,000 study participants. Response rates for the first and second mailings are averages from studies 1 and 2 (tables 2 and 3), and the response rate for mailing 3 (26 percent) was based on study 2. Costs were obtained from study 2 vendors and were $1.90 for mailing a questionnaire packet, including a cover letter, a 16-page questionnaire, and outgoing and return envelopes, as well as outgoing postage and packet assembly. Return mailing costs included only the cost of return postage by business reply mail ($1.28). Additional costs for including a pencil in the second mailing were $0.18 for the study-logo pencil and a postal charge for nonmachine reading when the pencil was included in the packet. There was no additional charge for packet assembly using our vendor, nor was there additional outgoing postage when the pencil was included. Application of these costs and response rates to a three-mailing strategy yielded an increased overall response rate (88 percent) for inclusion of the pencil in the second mailing as opposed to no pencil (84 percent), while costs were almost identical ($39,753 with the pencil vs. $39,698 without it).
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Our results are in contrast to those of three earlier studies of whether including a pen or pencil affects response rates in epidemiologic or health surveys in the general (non-health-professional) population. Aveyard et al. (5) tested whether sending a pencil and eraser with a questionnaire to smokers selected from health records would increase the response rate. Including a pencil and eraser led to a nonsignificant decrease in the response rate (odds ratio = 0.77, 95 percent confidence interval: 0.46, 1.29). In a follow-up mailing to persons who had recently completed baseline questionnaires and had blood drawn for the CLUE II Study, Hoffman et al. (6
) found that survey response was almost identical for persons sent a printed pencil with a mailed questionnaire (47 percent) and persons sent no incentive (45 percent). Etter et al. (7
) conducted a study with a three-way factorial design in which students, faculty, and staff of a Swiss university were sent a survey on smoking habits and were randomized to receive 1) a request to provide a saliva sample, 2) a ballpoint pen, and/or 3) an offer to participate in a lottery for a weekend trip. Overall, 59 percent of those randomized to receive the pen returned the questionnaire within 28 days, as opposed to 56 percent of those not sent a pen (p = 0.10). In related studies, Clark et al. (10
) found that inclusion of a pen did not increase response to a questionnaire sent to gynecologists, and Willimack et al. (11
) reported a nonsignificant 5 percentage point increase (81 percent vs. 76 percent; p = 0.29) when a gift-type ballpoint pen was sent with the recruitment letter for a face-to-face interview on social issues. It is not clear why we observed a large effect of including a pen or pencil in our two trials while these studies did not.
If, as we found, inclusion of a pen or pencil does increase response, it may be because it is viewed as an incentive or a reward. Numerous studies have been conducted on nonmonetary incentives designed to increase response rates. Two meta-analyses done in the early 1990s, each of which reviewed 12 (overlapping) studies, found that nonmonetary incentives improved response rates by 7.58 percentage points, on average (3, 4
). More recently, Edwards et al. (2
) reviewed 45 studies of nonmonetary incentives and reported that such incentives modestly increase response rates, on average (odds ratio = 1.19, 95 percent confidence interval: 1.11, 1.28; approximately a 5 percentage point increase for response rates in the range of 4060 percent). The nonmonetary incentives included pencils, pens, lottery tickets, coffee coupons, tea bags, and newspaper articles on the study. Our observed effect was much greater: 15 percentage points in one study and 19 percentage points in the other. This difference may result from our trials' being limited to nonresponders to a first mailing. Alternatively, we may have observed a stronger effect than the average effect in the other nonmonetary incentive trials because a pen or pencil is a facilitating factor as well as a token of appreciation. It may facilitate completing the questionnaire because the respondent does not have to search for a pen or pencil before beginning to fill out the survey, or perhaps some homes do not have a pen or pencil available. However, as we noted above, the trials most similar to ours did not find a significant effect of including a pen or pencil.
Advantages of this report are that it includes two independent trials, one for population-based study recruitment and one for a follow-up survey in a cohort study. Limitations are that both trials were based on lists of people who had participated previously in some way (persons who had undergone mammograms in New Hampshire and agreed to be contacted for research and persons who had completed a cohort baseline questionnaire 2 years earlier in the VITAL Study). Therefore, our response rates to the first mailing were high, and our results for the second mailing may not be generalizable to studies that have low response rates to a first mailing. Another limitation is that we only tested inclusion of a pen/pencil in the second mailing. Our rationale was that it would be too expensive to include a pen/pencil in the first mailing, which was to be sent to tens of thousands of people in the full studies. We also felt that a large percentage of those contacted would participate without an incentive, so the costs of the pen/pencil could be reserved for the nonrespondents. Our studies cannot answer the question of whether inclusion of a pen/pencil in a first mailing would increase response rates. Finally, our estimates of costs were specific to our findings on the effect of a pen/pencil on response rates and to the associated costs in one of our studies. Therefore, researchers may wish to conduct their own preliminary trial and cost estimation across all mailings before deciding whether to include a pen or pencil in some mailings.
In summary, this study suggests that inclusion of a study-logo pen or pencil in a second questionnaire mailing to nonrespondents is a cost-effective method of increasing response rates. Further studies of the effect on response rates and associated costs of including a pen or pencil in a first mailing are needed to understand whether that would be a more or less cost-effective approach.
![]() |
ACKNOWLEDGMENTS |
---|
Conflict of interest: none declared.
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|