Using Publicly Available Directories to Trace Survey Nonresponders and Calculate Adjusted Response Rates

Bart J. Harvey1,2 , Annette L. Wilkins3, Gillian A. Hawker1,3,4,5,6, Elizabeth M. Badley1,2,3, Peter C. Coyte1,5,6, Richard H. Glazier1,2,3,7,8, J. Ivan Williams1,2,6,9 and James G. Wright1,2,5,6,10,11

1 Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
2 Department of Public Health Sciences, University of Toronto, Toronto, Ontario, Canada.
3 Arthritis Community Research and Evaluation Unit, Toronto, Ontario, Canada.
4 Division of Rheumatology, Sunnybrook & Women’s College Health Sciences Centre, Toronto, Ontario, Canada.
5 Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada.
6 Institute for Clinical and Evaluative Sciences, Toronto, Ontario, Canada.
7 Department of Family and Community Medicine, St. Michael’s Hospital, Toronto, Ontario, Canada.
8 Inner City Health Research Unit, St. Michael’s Hospital, Toronto, Ontario, Canada.
9 Toronto Rehabilitation Research Institute, Toronto, Ontario, Canada.
10 Division of Orthopaedic Surgery, Hospital for Sick Children, Toronto, Ontario, Canada.
11 Department of Surgery, University of Toronto, Toronto, Ontario, Canada.

Received for publication December 2, 2002; accepted for publication May 15, 2003.


    ABSTRACT
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
In population-based surveys, sample lists are often out of date by the time data collection begins. Consequently, response rates, and the perceived validity of the survey, may be compromised by the unknowing inclusion of ineligible subjects. A strategy to address this issue is ascertainment of survey nonrespondents’ eligibility status, enabling post hoc adjustment of response rates. In 1995–1996, population surveys were carried out in two Ontario, Canada, communities. Despite intensive follow-up, the status of 8,949 (18.6%) of the 48,218 potential subjects in these surveys remained unknown. In response, 500 "unknowns" from each community were randomly selected for tracing by using publicly available telephone directories and, where applicable, city directories. These tracing efforts classified persons into one of three groups: "ineligible" (moved before the mailing), "true nonresponder" (present when the survey was mailed), and "remains unknown" (no directory listing found). Publicly available directories clarified the status of 76.0% of potential participants, reducing the proportion of "unknowns" from 18.6% to 4.6%. Applying the estimated proportions of "ineligibles" from each area resulted in response rates adjusted from 63.8% to 71.2% and from 72.8% to 74.9% in the survey areas. Publicly available directories were used to successfully trace the majority of survey nonresponders, thus strengthening confidence in the survey’s results.

contact tracing; data collection; directories; eligibility determination; nonrespondents


    INTRODUCTION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
In population-based survey research, most sample lists are out of date by the time data collection begins. Consequently, response rates may be reduced because ineligible subjects are included. This inaccuracy in response rates poses difficulties in assessing the validity of survey results (1, 2). Ideally, the sample lists would be corrected before data collection. Because this step is often not possible, an alternative is to make adjustments post hoc. Doing so requires ascertaining the eligibility status of survey nonrespondents.

In 1995–1996, population-based surveys were carried out in two Ontario, Canada, communities (3, 4): Toronto’s Borough of East York, an urban area, and Oxford County, a mostly rural area. The sample listing used was the Ontario Ministry of Finance’s 1994 residential tax records, which became available to the investigators in September 1995. Eligible respondents were those aged 55 years or older living in either area at the commencement of questionnaire mailing (November 1, 1995): 26,293 persons in East York and 21,925 in Oxford County. After two follow-up mailings and attempted telephone contacts were completed, 14,082 (53.6 percent) surveys were returned from East York and 14,369 (65.5 percent) from Oxford County (table 1).


View this table:
[in this window]
[in a new window]
 
TABLE 1. Status of participant surveys, Ontario, Canada, 1995–1996
 
Despite these follow-up efforts, the status of 5,451 respondents (20.7 percent) from East York and 3,498 (16.0 percent) from Oxford County remained unknown. Research (5) suggests that many nonresponses may be due to intended recipients having moved away and their mail not being identified as such and thus returned to the sender by the postal system. Furthermore, annual mobility rates for East York and Oxford County are about 15 percent (6, pp. 469 and 739; 7, pp. 913 and 1381). Because the group of 8,949 "unknowns" represented a large proportion of the sample (18.6 percent), more extensive tracing efforts were applied post hoc in 1998 to identify whether any of these persons had actually moved prior to the survey’s mailing and therefore would have been ineligible. Because data collection had concluded by the time tracing was considered, these efforts were undertaken solely to assist with interpretation of survey results by providing a basis upon which further adjusted response rates might be calculated. The process was also expected to provide a better understanding of the utility of tracing using publicly available directories.


    MATERIALS AND METHODS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
The study was conducted in Oxford County and East York because these locations were ranked in the top and bottom quintiles, respectively, regarding rates of hip and knee arthroplasty in the province of Ontario’s 49 geographic areas. In the study’s first phase, a brief questionnaire was mailed to all persons aged 55 years or older in the two communities. The questionnaire was designed to identify persons with moderately severe hip or knee complaints who would be invited to participate in the study’s second phase. The names and addresses of eligible persons were obtained from the Ontario Ministry of Finance’s 1994 residential tax records.

Those not responding after two additional mailings were telephoned in accordance with National Center for Health Statistics methods (8). Because telephone numbers were not included in the Ministry of Finance database, current telephone directories were used to search for each nonrespondent’s telephone number. As shown in table 1, these follow-up efforts clarified the status of 20,842 persons (79.3 percent) in East York and 18,427 (84.0 percent) in Oxford County, including a large number identified as being either ineligible or unable to complete the survey. Eliminating these persons resulted in adjusted response rates of 63.8 percent (14,082/22,063) for East York and 72.8 percent (14,369/19,735) for Oxford County.

While a wide range of resources, such as motor vehicle, real estate, utility company, and death records, have been used to trace survey nonrespondents’ whereabouts (9, 10), we chose to use relevant past editions of telephone and city directories, readily available in print or microfiche at local libraries. This approach involved searching telephone directories by last name and, where applicable, city directories by street address.

Because resource constraints precluded tracing all nonresponders, 500 "unknowns" from each community were randomly selected for more extensive tracing using publicly available directories. This number was chosen so that the proportions would be estimated with a precision of at least ±5.0 percent (11), providing sufficient precision to reapply the estimated proportions to the entire group of unknowns in each area.

Tracing involved a search of each area’s 1995 telephone directory to identify a listing with the last name and address of each person sought. If such a listing was found, the 1996 telephone directory was also searched. If a consistent listing was found in 1996, the person was considered a true nonresponder, having resided at the study address when the survey was mailed. If a corresponding listing within the study community could not be found for that person in the 1996 telephone directory (but was present in 1995), the person was deemed ineligible for the survey, having moved before the survey was mailed. To confirm each apparent move, the 1996 telephone directory was also examined to determine whether the person was listed at a new address.

For those communities for which city directories were available (i.e., all of East York and the city of Woodstock in Oxford County), the 1995 city directory was searched to identify the listing corresponding to the address of each person sought. If the intended recipient could not be identified by using the 1995 city directory, then early editions of the directory (back to 1989) were also searched. When a study participant was identified by using a city directory, the listing for that address in the following year’s edition of the city directory was reviewed. If a new occupant could be identified at the applicable study address, an attempt was made to identify his or her listing in the applicable telephone directory, thus confirming that the telephone at the address remained publicly listed, providing further evidence that the study subject had not unlisted the telephone number but had in fact actually moved.

If listings corresponding to a study participant and his or her address could not be identified in the telephone or city directories, the person’s survey status remained "unknown."


    RESULTS
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Detailed address information was available for all (n = 5,451) of the East York unknowns; however, only more general information such as rural route, post office box, and/or "General Delivery" was available for 32.8 percent (1,146/3,498) of those from Oxford County. Furthermore, while city directories were available for all of East York and for each of the years searched, comparable Oxford County availability existed for Woodstock only, where 42.7 percent (1,493/3,498) of Oxford County unknowns lived.

While the 500 randomly selected East York unknowns were being traced, the status of 23 was confirmed through sources unrelated to the tracing procedures. As such, the East York sample ultimately consisted of the remaining 477 persons. The results of the tracing efforts differed for the two areas (table 2). Although similar proportions in both East York and Oxford County remained "unknown" (26.4 percent and 21.6 percent, respectively), unknowns in East York were more likely to be classified as "ineligible at baseline" (41.9 percent vs. 16.0 percent) and less likely to be classified as "true nonresponders" (31.7 percent vs. 62.4 percent).


View this table:
[in this window]
[in a new window]
 
TABLE 2. Results of telephone and city directory tracing efforts, Ontario, Canada, 1995–1996
 
Because of the differential availability of city directories and detailed addresses, further between-community assessments were carried out. Comparing the Oxford County subset that was similar to the East York group—those for whom detailed address information was accessible and city directories were available—resulted in a similar proportion of "ineligible at baseline" being identified in each area (table 2).

Applying the proportions found "ineligible at baseline" to the total number of unknowns in each area resulted in a further 2,284 (41.9 percent of 5,451) and 560 (16.0 percent of 3,498) persons being estimated as ineligible in East York and Oxford County, respectively. These estimates resulted in readjusted response rates of 71.2 percent (14,082/19,779) for East York and 74.9 percent (14,369/19,175) for Oxford County.


    DISCUSSION
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
This study’s results demonstrate that publicly available directories can be used to successfully trace the majority of survey nonrespondents, identifying them as either ineligible or true nonresponders. The study also illustrates the utility of the process, enabling calculation of adjusted response rates by using the estimated proportions of "ineligibles."

The use of tracing to determine respondent eligibility has not, to our knowledge, been described previously. These efforts, while requiring significant resources to complete effectively, were worthwhile because they provided a more complete understanding of study nonresponse and, furthermore, enabled presentation of more informative, adjusted survey response rates.

This study was possible only because of the existence of and access to the 1994 Ministry of Finance database. However, it was important to assess the accuracy of the Ministry of Finance listing of those aged 55 years or older in each study area. A review of 1991 and 1996 census data (7, pp. 909 and 1377; 12, pp. 296 and 453) revealed that the numbers of persons aged 55 years or older listed in the Ministry of Finance database for East York and Oxford County (26,293 and 21,925, respectively) were similar to the corresponding census-derived estimates (26,397 and 21,975). In addition, the age-sex distributions of the persons listed in the Ministry of Finance database were also found to be similar to the census data.

Although tracing only a random subset of nonresponders is more efficient and feasible than tracing the entire sample, it does provide proportions estimated with imprecision. However, the impact of this imprecision can be evaluated by conducting sensitivity analyses. For example, in our study, applying the lower limit of the 95 percent confidence intervals for the proportion who were "ineligible at baseline" (37.5 percent for East York and 12.8 percent for Oxford County) revealed "worst-case" adjusted response rates of 70.3 percent (14,082/20,019) for East York and 74.5 percent (14,369/19,287) for Oxford County—both satisfying the recommended minimum survey response rate of 70 percent (1).

The ability to complete tracing efforts such as these is greatly aided by sample lists that provide accurate, complete, and thorough baseline information concerning each person in the study sample. In addition, such efforts are possible only when publicly available telephone and city directories are available—ideally, for all years and persons studied. The importance of detailed address information and available city directories is highlighted by data from Oxford County, where the overall mobility rate (16.0 percent) was lower than the rate for those with detailed addresses and for whom city directories were available (42.2 percent). Without detailed addresses and available city directories for all of Oxford County, it was impossible to determine whether this difference was real or simply the result of having less information for 344 of the Oxford County unknowns.

Because moving is only one reason for someone no longer being listed in a telephone directory, alternative explanations were also examined. First, in areas served by more than one telephone provider, people may have simply changed providers. However, during the time of this survey, Bell Canada held a monopoly to provide residential telephone services. In addition, while cellular telephones were in use in 1996, we expect that very few study subjects (i.e., those aged 55 years or older) would have replaced their residential telephone with cellular service.

A second reason that persons may no longer be listed in the telephone directory is if their listing is taken over by another family member. In this situation, however, the telephone number would be expected to remain unchanged to avoid telephone disconnection and connection fees. Our tracing efforts did not identify any such listing name changes.

A third reason that persons may no longer be listed in the telephone directory is if they change to an unlisted number. Several pieces of evidence argue against such an explanation in this study, however. First, according to city directory listings, the traced "unknowns" had been listed publicly at their address for an average of 8 years. Therefore, it is unlikely that many of these persons would have simultaneously switched to unlisted numbers en mass in 1996. Second, while telephone contacts resulted in the completion of a large number of surveys (table 1), no recorded messages identifying the "number unlisted at the subscriber’s request" were encountered during this follow-up process. In addition, for 90 percent of the unknowns, either a telephone number could not be located or, when telephone contact was attempted, the number was found to be either out of service or incorrect. Third, 140 of the tracing-identified moves were confirmed by identifying either a new address for the study subject or the telephone listing of a new occupant at the study address: 45/70 (64.3 percent) in Oxford County’s city of Woodstock and 95/200 (47.5 percent) in East York. Because of less specific city directory listings for apartments, a larger proportion of moves from detached homes were confirmed in both areas: 26/30 (86.7 percent) in Woodstock and 40/56 (71.4 percent) in East York.

Given sufficient person-power, these tracing efforts would be more ideally carried out in "real-time," with data collection still under way, rather than post hoc to better understand and clarify the survey’s response rate. Of course, for this study, that would have involved tracing nearly 9,000 persons instead of just the 977 who were selected randomly.

Despite requiring a significant investment of resources, tracing efforts can provide, as demonstrated by this study, valuable information concerning a survey, particularly quantifying deficiencies in the sample lists caused by inclusion of ineligible persons. This information then provides a basis for estimating readjusted response rates and serves to strengthen understanding of and confidence in the survey’s results.


    ACKNOWLEDGMENTS
 
This work was supported through grants from the Medical Research Council of Canada (MT-12919), The Arthritis Society of Canada (97-083), Physicians’ Services Foundation Incorporated (95147), The Ontario Ministry of Health (Health System Link funding to the Arthritis Community Research and Evaluation Unit), The Canadian Orthopaedic Foundation, and the University of Toronto Dean’s Fund (00026896). Drs. Hawker and Wright receive salary support from the Canadian Institutes of Health Research (CIHR) as CIHR Investigators, and Dr. Coyte receives salary support from both the Canadian Health Services Research Foundation (CHSRF) and the CIHR as a CHSRF/CIHR Health Services Chair.


    NOTES
 
Correspondence to Dr. Bart J. Harvey, Department of Public Health Sciences, University of Toronto, 4032-12 Queen’s Park Crescent West, Toronto, Ontario, Canada M5S 1A8 (e-mail: bart.harvey{at}utoronto.ca). Back


    REFERENCES
 TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Jones J. The effects of non-response on statistical inference. J Health Soc Policy 1996;8:49–62.[Medline]
  2. Dillman DA. Mail and Internet surveys: the tailored design method. 2nd ed. New York, NY: John Wiley & Sons, 2000:194–201.
  3. Hawker GA, Wright JG, Coyte PC, et al. Differences between men and women in the rate of use of hip and knee arthroplasty. N Engl J Med 2000;342:1016–22.[Abstract/Free Full Text]
  4. Hawker GA, Wright JG, Coyte PC, et al. Determining the need for hip and knee arthroplasty: the role of clinical severity and patients’ preferences. Med Care 2001;39:206–16.[CrossRef][ISI][Medline]
  5. Sandler RS, Holland KL. Fate of incorrectly addressed mailed questionnaires. J Clin Epidemiol 1990;43:45–7.[ISI][Medline]
  6. Statistics Canada. Profile of census divisions and subdivisions in Ontario, part B. Ottawa, Ontario, Canada: Industry, Science and Technology Canada, 1994. (1991 census of Canada, catalogue no. 95-338).
  7. Statistics Canada. Profile of census divisions and subdivisions in Ontario. Ottawa, Ontario, Canada: Industry Canada, 1999. (1996 census of Canada, catalogue no. 95-187-XPB).
  8. Massey JT. Overview of the National Health Interview Survey and its sample design. Vital Health Stat 2 1989;Aug:1–5.
  9. Nordberg PM. Leave no stone unturned. Am J Epidemiol 1992;136:1160–6.[Abstract]
  10. Hunt JR, White E. Retaining and tracking cohort study members. Epidemiol Rev 1998;20:57–70.[ISI][Medline]
  11. Glantz SA. Primer of biostatistics. 4th ed. New York, NY: McGraw-Hill, 1997:201–4.
  12. Statistics Canada. Profile of census divisions and subdivisions in Ontario, part A. Ottawa, Ontario, Canada: Supply and Services Canada, 1992. (1991 census of Canada, catalogue no. 95-337).




This Article
Abstract
FREE Full Text (PDF)
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Disclaimer
Request Permissions
Google Scholar
Articles by Harvey, B. J.
Articles by Wright, J. G.
PubMed
PubMed Citation
Articles by Harvey, B. J.
Articles by Wright, J. G.