Contacting Controls: Are We Working Harder for Similar Response Rates, and Does It Make a Difference?
A. Rogers,
M. A. Murtaugh ,
S. Edwards and
M. L. Slattery
From the Health Research Center, Department of Family and Preventive Medicine, School of Medicine, University of Utah, Salt Lake City, UT.
Received for publication October 16, 2003; accepted for publication February 4, 2004.
 |
ABSTRACT
|
---|
Although obtaining high response rates is critical to epidemiologic studies, effort to achieve response rates is undocumented. The authors used three population-based case-control studies conducted in Utah between October 1991 and February 2003 to examine effort required for both initial contact and determination of final status. Differences in lifestyle characteristics between easy- or more-difficult-to-interview female controls were evaluated. Letter, phone, and in-person contacts were recorded to determine contact effort. Regarding effort required to achieve a final outcome, the number of contacts increased from eight to 14 over the 12-year study period. Compared with those in study A (conducted in 19911994), controls in studies B and C were twice as likely to require seven or more phone calls and controls in study B were twice as likely to require one or more in-person visit. Hispanic controls in study C were more likely than non-Hispanic White controls to receive an in-person visit and a noncontact letter. Compared with those more difficult to contact, those easy to contact were more likely to be overweight and less likely to have a family history of cancer. The amount of effort required to achieve similar or slightly lower response rates increased over time. This finding may in part depend on demographic characteristics of the population studied.
data collection; sampling studies; selection bias
Abbreviations:
Abbreviations: CI, confidence interval; OR, odds ratio.
 |
INTRODUCTION
|
---|
Response rates provide an indication of the generalizability of study results to the target population. Evidence is mixed with regard to change in response rates (15). In an examination of several large-scale studies, the overall trend was toward lower participation, but rates varied among studies, making it difficult to forecast response rates (6). Less information is available on the amount of effort and time required to achieve high response rates (6). Some studies have defined contact effort as time (79) or total number of phone calls (10, 11) or as phone rather than mail contact (1214). We found only one study (15) that reported using the total number of contact attempts (i.e., phone, mail, in-person visits) to define contact effort and no studies examining contact effort to the initial contact or final outcome for all controls identified, including those who did not participate.
The purpose of this study was to examine the effort required to contact female controls over time; we used data from three population-based case-control studies in the same geographic location with similar recruitment protocols. We examined whether effort increased in 1) making the initial contact and 2) getting to a final outcome (i.e., interviewed, refused, ineligible, etc). We also investigated whether demographics and study exposure variables differed by effort needed for initial contact and final outcome.
 |
MATERIALS AND METHODS
|
---|
Data came from the Utah center of three multicenter, population-based, case-control studies using similar protocols. Participants were recruited for an in-person interview, which was usually conducted in the participants home; took approximately 2 hours to complete; and asked for detailed information about diet, activity, reproductive history, and family history of any cancer. Controls were matched to cases by 5-year age groups and were randomly selected from drivers license and Health Care Financing Administration lists. The Generally Useful Ethnic Search System (GUESS) software program (16) was used to assign expected ethnicity to controls in studies A and B when ethnicity was not known. Missing data were due to participant refusal or an answer of "dont know." Analyses were completed by using 1999 SAS statistical software (SAS Institute, Inc., Cary, North Carolina).
In all studies, controls were randomly selected (study A: October 1, 1991September 30, 1994 (17); study B: October 1, 1997November 31, 2001 (18); and study C: January 1, 2001present) by using drivers license lists for controls aged 64 years or less and Centers for Medicare and Medicaid Services lists for controls 65 years of age or more. In study A, participants of any race/ethnicity except Asian and Native American were eligible; in study B, women of all races/ethnicities were eligible; and, in study C, eligible participants were White (non-Hispanic and Hispanic) or Native American. Study A restricted controls to counties in northern Utah, whereas studies B and C recruited statewide. In addition to being matched on 5-year age categories, controls in study C were matched to cases by Hispanic ethnicity. A letter explaining the study was sent to selected controls; 1 week later, initial contact attempts were made. If needed, a noncontact letter was sent requesting a phone number and time for contact. In-person visits were made to the residences of nonresponders and from which letters were returned.
Phone and in-person contact information including date, time of day, day of week, and outcome (e.g., no answer or no one home, answering machine, busy signal, etc.) was recorded on the respondent information sheet. Contact attempts up to the initial contact and the final outcome (i.e., interviewed, refused, etc.) were summed to create initial and final contact counts. In this paper, we include the numbers of attempts to contact 43 controls (17 from study A, 12 from study B, and 14 from study C) who were never reached successfully. In studies A and B, data were used for all eligible controls. In study C, we used data for controls for whom a final outcome was entered before February 1, 2003. One control from each study was excluded because the respondent information sheet was missing (table 1).
We used study A as the referent group to set cutpoints for identification of controls above the 90th percentile regarding the number of contacts made and applied these cutpoints to studies B and C. Difficult to make an initial contact was defined as four or more phone calls, one or more in-person visit, or one or more noncontact letter up to the initial contact date. Difficult to reach a final outcome was classified as seven or more phone calls, one or more in-person visit, or one or more noncontact letter up to the final contact date. Overall difficulty in making initial or final contact with a potential participant was defined as making more than four or seven calls, respectively; making one or more in-person visit; and/or sending one or more noncontact letter.
In each study, interviewed participants categorized as above the 75th percentile regarding total number of contacts (whether by phone, noncontact letter, and/or in-person visit) were defined as difficult to interview. Intermediate and easy-to-interview participants were very similar, so we combined them into one group to increase the precision of our point estimates. We first examined studies individually and then combined studies and used the same study-specific criteria to define the difficult-to-interview participants.
Univariate analyses were conducted by using a chi-square test to compare contact effort across studies, compare participants easy and more difficult to contact, and compare contact effort by ethnicity in study C. Logistic regression was used to conduct multivariate comparisons of contact effort between ethnicities in study C and to study exposures between interviewed participants easy and more difficult to contact (adjusted for age in studies A and B and for age and race/ethnicity in study C). A polytomous logistic regression model was used to estimate odds ratios and 95 percent confidence intervals of point estimates across studies by using study A as the referent group and adjusting for age and race.
 |
RESULTS
|
---|
Controls in study A were older than those in studies B and C (table 2). There were more Hispanic controls in study C than in studies A and B. More women in study C were ineligible to participate, most commonly because of race/ethnicity (table 1). The number of participants with unlisted or nonpublished phone numbers increased over time. Interviewed participants in study B were slightly more educated than those in studies A or C.
The numbers of phone calls, in-person visits, and noncontact letters to initial contact and final outcome increased over time (table 3). After we adjusted for age and race/ethnicity, compared with study A, more noncontact letters were sent in study B (odds ratio (OR) = 1.71, 95 percent confidence interval (CI): 1.12, 2.62) and more phone calls were made in study C (OR = 1.58, 95 percent CI: 1.01, 2.48) until the initial contact. To reach a final outcome, compared with study A, controls in study B were more likely to require seven or more phone calls (OR = 1.93, 95 percent CI: 1.22, 3.04) and one or more in-person visit (OR = 2.03, 95 percent CI: 1.28, 3.22). In study B, controls were 1.69 (95 percent CI: 1.11, 2.58) times more likely than those in study A to have one or more noncontact letter sent. Compared with those in study A, controls in study C were twice as likely to require seven or more phone calls (OR = 2.09, 95 percent CI: 1.29, 3.38) to reach a final outcome. Greater effort was expended to arrive at a final outcome for participants in studies B (OR = 1.61, 95 percent CI: 1.15, 2.25) and C (OR = 1.45, 95 percent CI: 1.00, 2.09). Compared with participants easier to contact, approximately 10 percent of participants difficult to contact were easier to interview. The total number of contacts needed to reach a final outcome for 90 percent of the control population increased from eight in study A to 14 in study C over this 12-year time period.
View this table:
[in this window]
[in a new window]
|
TABLE 3. Contact effort required up to initial contact and final outcome for female controls in three case-control studies in Utah, 20012003
|
|
After adjusting for age, we found that to make an initial contact, interviewers were 5.36 (95 percent CI: 2.89, 9.93) times more likely to make an in-person visit and 4.08 (95 percent CI: 2.48, 6.73) times more likely to send a noncontact letter to Hispanic controls than to non-Hispanic controls. Contact effort to a final outcome was similar. Overall, it was 3.10 times more difficult to arrive at a final outcome for Hispanic controls than for non-Hispanic controls in study C.
Participants who were more difficult to contact and interview were more likely to be unmarried, be aged 5064 years, and have less than a college education. After adjusting for age, race/ethnicity, and study, we found that compared with participants difficult to interview, those easier to interview were 2.34 (95 percent CI: 1.36, 4.02) times more likely to have a body mass index (weight (kg)/height (m)2) of 2529 than of less than 25. In addition, those who were easier to interview were roughly 40 percent less likely (OR = 0.64, 95 percent CI: 0.42, 1.00) to report having a first-degree relative with cancer. No differences were found in smoking status; use of nonsteroidal antiinflammatory agents; cancer screening; alcohol intake; red meat, fruit, and vegetable or energy intake; or vigorous leisure activity.
 |
DISCUSSION
|
---|
Our analysis showed that to achieve the same, or lower, response rate between 1991 and 2003, the total number of contacts needed to initially contact 90 percent of the female controls rose from four to eight. After initial contact, it took more contacts to achieve a final outcome. The number of phone calls, noncontact letters, and in-person visits required to reach a final outcome for 90 percent of the participants rose from eight to 14. These differences were more striking for Hispanic and non-Hispanic White controls.
Of interest is the benefit obtained from additional contacts in terms of maintaining the response rate. In study A (19911994), a final outcome was reached for 75 percent of the population with five contacts, and a final outcome was achieved for 95 percent of the population with 10 contacts. In 19972002, eight contacts were needed for 75 percent and 17 contacts were required for 95 percent of the population to reach a final outcome; in 20012003, the numbers of contacts rose to nine and 20, respectively. Some of these differences in the most recent study could stem from different ethnic backgrounds of study participants.
In our study, greater effort was required to contact and to interview Hispanic women than non-Hispanic White women. More noncontact letters were sent and in-person visits made to Hispanic than to non-Hispanic White women, perhaps because of the higher prevalence of unlisted or disconnected phone numbers among the Hispanic women. For most of the risk factors we examined, we found little evidence of potential bias if we had excluded the difficult-to-interview controls. However, those who required more effort to interview were more likely than those easier to interview to report a family history of cancer in first-degree relatives. In contrast, interview data from six population-based case-control studies indicated that female controls who responded early (<1 month from contact to interview) were more likely than women interviewed more than 6 months after postcard contact to report having a first- or second-degree female relative with a history of cancer (7). Time to participation was not considered in our quantification of effort to contact, and we considered male and female first-degree relatives history of any cancer. Whether these differences explain the disparate findings is unknown. In studies A and B, the initial letter sent to participants stated that we were conducting a study on dietary and physical activity, but there was no mention of cancer. However, in study C, the letter indicated that we were studying breast cancer, and we found that women with a family history of breast cancer were more likely to participate than those without a family history.
Unexpectedly, participants who were less difficult to interview were more likely than those difficult to interview to be overweight (body mass index, 2529). It is not immediately clear whether this difference in body mass index was spurious or was related to some other characteristic of the participants. We could find no previous reference in the literature to a difference in body weight when comparing effort to recruit participants.
While some previous studies also found little evidence of selection bias (10, 11, 13, 14), others found differences in health indicators based on time of response or difficulty in locating participants. Compared with early responders, late responders (>5 months) to a mailed survey of retirement community residents were more likely to perceive a deterioration in mental function and have difficulty coping with their social setting (9). In a cohort of veterans, prevalence of known risk factors for diminished health status was increased among those difficult-to-locate interviewed veterans (8). Difficult-to-find veterans and late-responding female controls from six case-control studies were more likely to smoke than easier-to-locate or earlier-responding participants (7, 8). In a study of demographic characteristics of those with current psychiatric disorders, only alcohol disorder was predictive of difficulty in contacting participants and a higher refusal rate (15). Compared with late responders, women who responded quickly to a mail/phone survey investigating whether socioeconomic differences explain observed differences in health or use of health care services were more likely to use health care services (12). Thus, the implications of these findings regarding potential response bias remain complex and seemingly situational.
Our results documenting an increasing effort to obtain similar participation rates have implications for study design and implementation. Over a 12-year period, the number of contacts required to achieve a final outcome for 90 percent of the population rose from eight to 14. This increase is similar to the reported doubling of phone calls needed to complete a phone interview from 1979 to 1996 (19). More contacts also were labor intensive (in-person visit vs. phone). However, those who were easy versus difficult to recruit and interview differed little regarding studied exposures.
 |
ACKNOWLEDGMENTS
|
---|
This study was funded by grants CA48998 and CA78682 from the National Cancer Institute.
The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official view of the National Cancer Institute.
The authors acknowledge James Bryner for creating the database; Karen Curtin, Cempaka Martial, and Richard Holubkov for advice regarding SAS statistical software; Steve Alder and Richard Holubkov for statistical advice; Roger Edwards for technical support; Leslie Palmer for study management; and the study interviewers for data collection.
 |
NOTES
|
---|
Correspondence to Dr. Maureen A. Murtaugh, Health Research Center, Department of Family and Preventive Medicine, School of Medicine, University of Utah, 375 Chipeta Way, Suite A, Salt Lake City, UT 84108 (e-mail: mmurtaugh{at}hrc.utah.edu). 
 |
REFERENCES
|
---|
- Altrosic B, Bates N, Burt G, et al. Nonresponse in US government household surveys: consistent measures, recent trends, and new insights. J Official Stat 2001;17:20926.
- Gorey KM, Trevisan M. Secular trends in the United States black/white hypertension prevalence ratio: potential impact of diminishing response rates. Am J Epidemiol 1998;147:959. [Abstract]
- Eichner JE. Invited commentary on "secular trends in the United States black/white hypertension prevalence ratio: potential impact of diminishing response rates." Am J Epidemiol 1998;147:1002.[ISI]
- Olson SH. Reported participation in case-control studies: changes over time. Am J Epidemiol 2001;154:57481.[Abstract/Free Full Text]
- Stang A, Ahrens W, Jockel KH. Control response proportions in population-based case-control studies in Germany. Epidemiology 1999;10:1813.[ISI][Medline]
- Groves R, Couper M. Nonresponse in household interview surveys. New York, NY: John Wiley & Sons, Inc, 1998.
- Voigt LF, Koepsell TD, Daling JR. Characteristics of telephone survey respondents according to willingness to participate. Am J Epidemiol 2003;157:6673.[Abstract/Free Full Text]
- Decouflé P, Holmgreen P, Calle EE, et al. Nonresponse and intensity of follow-up in an epidemiologic study of Vietnam-era veterans. Am J Epidemiol 1991;133:8395.[Abstract]
- Paganini-Hill A, Hsu G, Chao A, et al. Comparison of early and late respondents to a postal health survey questionnaire. Epidemiology 1993;4:3759.[ISI][Medline]
- Kristal AR, White E, Davis JR, et al. Effects of enhanced calling efforts on response rates, estimates of health behavior, and costs in a telephone health survey using random-digit dialing. Public Health Rep 1993;108:3729.[ISI][Medline]
- Bull SB, Pederson LL, Ashley MJ, et al. Intensity of follow-up: effects on estimates in a population telephone survey with an extension of Kishs (1965) approach. Am J Epidemiol 1988;127:55261.[Abstract]
- Brambilla DJ, McKinlay SM. A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. Am J Epidemiol 1987;126:96271.[Abstract]
- Vollmer WM, Osborne ML, Hertert S, et al. Recruiting hard-to-reach subjects: is it worth the effort? Control Clin Trials 1994;15:1549.[CrossRef][ISI][Medline]
- Siemiatycki J, Campbell S. Nonresponse bias and early versus all responders in mail and telephone surveys. Am J Epidemiol 1984;120:291301.[Abstract]
- Cottler LB, Zipp JF, Robins LN, et al. Difficult-to-recruit respondents and their effect on prevalence estimates in an epidemiologic survey. Am J Epidemiol 1987;125:32939.[Abstract]
- Buechley R. Generally useful ethnic search system: GUESS. Albuquerque, NM: University of New Mexico, 1976.
- Slattery ML, Potter J, Caan B, et al. Energy balance and colon cancerbeyond physical activity. Cancer Res 1997;57:7580.[Abstract]
- Slattery ML, Edwards SL, Ma KN, et al. Physical activity and colon cancer: a public health perspective. Ann Epidemiol 1997;7:13745.[CrossRef][ISI][Medline]
- Curtin R, Presser S, Singer E. The effects of the response rate changes on the index of consumer sentiment. Public Opinion Q 2000;64:41328.[Abstract/Free Full Text]