1 Department of Thoracic Medicine, Institute of Medicine, University of Bergen, Bergen, Norway.
2 Centre for Clinical Research, Haukeland University Hospital, Bergen, Norway.
3 Section for Medical Statistics, University of Bergen, Bergen, Norway.
Received for publication May 16, 2002; accepted for publication September 30, 2002.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
asthma; data collection; epidemiologic methods; health care; interviews; postal service; questionnaires; signs and symptoms, respiratory
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
To investigate the effect of increasing survey efforts on results, we performed a large-scale, population-based survey on respiratory health by using a postal questionnaire with a telephone and cellular phone follow-up. We also conducted a systematic literature review of studies of nonresponse bias in respiratory epidemiology.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Fieldwork
A postal questionnaire was mailed to each subject. After 3 weeks and 8 weeks, nonresponders were sent a reminder letter. Responders were classified as initial responders and responders to the first and second reminder letters.
A telephone follow-up was initiated for a 25 percent random sample of postal nonresponders. Telephone numbers for the follow-up sample were located by using a direct mail firm (DM Huset, Oslo, Norway), requiring a match on name, date of birth, and address. Six months after the first mailing, computer-assisted telephone interviews were performed by trained operators. Nine months after the study started, nonresponders without a telephone directory entry but with a cellular phone were located from the customer databases of the cellular phone providers. A similar cellular telephone interview was conducted.
Questionnaires were mailed, responses were processed, and telephone interviews were performed by the Interview Office of Statistics Norway.
Questionnaire
The questions regarding symptoms were taken from a modified translation of the British Medical Research Council on Chronic Bronchitis (24, 25). A one-sheet questionnaire designed for self-completion was used; it contained 45 questions to be answered by all subjects on 18 respiratory symptoms, seven respiratory and cardiac diagnoses, and 20 risk factors. Smokers were defined as daily smokers at the time of the study. Former smokers were persons who had smoked daily but had stopped (26). The study was approved by the Norwegian Data Inspectorate.
Statistical methods
Responders were categorized as initial responders, responders to the first reminder letter, responders to the second reminder letter, and responders to the telephone-based follow-up. These categories represented "survey effort." Prevalences and cumulative prevalences of symptoms or diagnoses by survey effort were computed, with 95 percent confidence intervals. A two-sided p value of less than 5 percent was taken as statistically significant. Equality between groups was investigated by using analysis of variance, ordered logistic regression, or logistic regression, as applicable. All statistical tests were of heterogeneity, not trend. To control for possible confounding of prevalence estimates, multiple logistic regression was performed (27). Age was entered as a cubic spline with knots at 15, 30, 50, and 70 years (28). Smoking habits was modeled as pack-years. Pack-years was modeled as a linear term for current smokers only. The adequacy of doing so was checked by using generalized additive models (29) with scatter-plot smoothing terms with 4 degrees of freedom for pack-years and age. Changes in the following risk factors with survey effort were assessed with interactions: gender, occupational exposure to dusts or gases, and pack-years. This analysis also adjusted for age, county, and educational level. A joint test that any interaction was significant was performed, and those significant at p < 0.10 were considered further. In addition, the odds ratios of the risk factors for all responders were computed. A missing response was defined as a selection of "dont know" or no response to a given question. Subjects for whom answers were missing in an analysis were excluded from that analysis. All calculations accounted for the sampling in the follow-up by using weighting and robust standard errors. Adjusted Wald tests were used to decide on the significance of terms. All data were analyzed by using Stata 7.0 software (30).
Literature search
A systematic literature search was performed to find studies of nonresponse bias in respiratory epidemiology. The following databases were searched in early 2002: MEDLINE (Bethesda, Maryland), CINAHL (Glendale, California), and the Cochrane databases through Ovid (New York, New York; http://www.ovid.com/). We also searched the database Science Citation Index Expanded (http://wos.isiglobalnet.com/) by using the phrase (("nonresponse" or "response" or "nonresponder" or "responder") and ("epidemiology" or "health surveys") and (pulm* or respirat* or lung*)). Studies that cited Drane et al. (31, 32) were identified by searching the citation database Science Citation Index Expanded. Matching references were retrieved and were screened on the basis of title and abstract. Only those studies of the general population in a defined geographic area on respiratory symptoms and/or diagnoses with at least a 10-year age span in the interval 1570 years were included. Reference lists of retrieved articles were also searched.
For simplicity purposes and to enable a conservative comparison of response rates across studies, response rates were recomputed as the ratio of the number of answers to the number of subjects who had not yet answered at that stage, regardless of whether they had received the letter, had a telephone, were dead, or had moved. Some numbers were calculated from percentages or read from graphs. The total response rate was recalculated with weighting for possible subsampling in the follow-up, when available.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The cumulative response rates after the first mailing, the first reminder letter, the second reminder letter, and the telephone follow-up were 42.7 percent, 60.7 percent, 68.3 percent, and 79.9 percent, respectively. The respective response rates in Oslo were 41.8 percent, 30.1 percent, 18.5 percent, and 34.6 percent; those in Hordaland were higher at 46.4 percent, 36.8 percent, 24.0 percent, and 46.6 percent for each successive phase of follow-up.
Willingness to respond was associated with age, gender, educational level, smoking habits, and occupational exposure to dusts or gases univariately (table 1). Men responded later than women. Age and educational level decreased with increasing contact effort. Current smoking and occupational exposure were more prevalent and pack-years increased with increasing contact effort.
|
The crude prevalences of symptoms and diagnoses by survey effort are shown in table 2. All symptoms were associated with survey effort except for attacks of breathlessness, dyspnea when walking two stairs, and dyspnea at rest. There was no uniform tendency toward an increase or decrease in prevalence with survey effort. Note in particular the large change for morning cough in responders to the telephone follow-up. No association for diagnoses was found.
|
|
|
Cost and time were substantial for the follow-up. The second reminder letter increased the cost of the survey by 25 percent. The telephone follow-up increased the cost of the survey by an additional 14 percent and netted 741 responses, while data collection time increased from 4 months to 12 months.
The literature search netted 305 hits in the electronic databases, of which 11 were included (22, 23, 3341) (table 4).
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Previous surveys using exactly the same postal survey methods had response rates of 89 percent in Oslo in 1972 (42) and 90 percent in Hordaland in 1985 (23). This decrease in response rates may reflect a temporal trend (43), although solid long-term data on response rates indicate that this trend varies with country and the circumstances of the survey (44). There may be reasons other than a simple decline in willingness to respond (4). The introductory text was longer because of regulatory and ethical concerns, and the questionnaire itself was slightly longer. The previous surveys were executed locally and were mailed in a University of Bergen envelope. The present survey was performed with the aid of Statistics Norway and was sent in a Statistics Norway envelope.
Oslo had a response rate 412 percent lower than that for Hordaland at each stage of survey follow-up. The survey methods were exactly the same in both counties. A number of sociodemographic differences between Oslo and Hordaland might explain the result. Oslo is the capital of Norway and is the undisputed economic and political center. It is a wholly urban area of 500,000 persons situated in a greater metropolitan area of about 1 million inhabitants (out of 4.5 million inhabitants in Norway as a whole). Hordaland is the center of coastal western Norway, a mixed urban and rural area with the city of Bergen (250,000 inhabitants) at its center (45). Gross income is 20 percent higher in Oslo than in Bergen (45).
Oslo has a large population of foreigners (foreign born or born to one or both foreign parents), constituting 19 percent of the population, whereas the figure is only 6.4 percent in Bergen (46). In the 2000 Norwegian census, the response rate was as low as 56 percent among those from Asia, Africa, and Latin America. When we reran the analyses separately by study area, the effects were of the same magnitude and direction.
We identified a telephone number for only 70 percent of the follow-up sample, whereas telephone coverage is universal in Norway. This figure was probably due to our conservative telephone book look-up algorithm. However, we then found a cellular telephone number for as many as 35 percent of those for whom we could not identify an ordinary telephone number.
For both the adjusted and unadjusted estimates, it appeared that the changes among telephone responders were driving most of the associations. Telephone responders had a lower risk of symptoms. The use of a telephone interview instead of a postal questionnaire may explain this finding. However, in an earlier study (47), we found few differences between the postal and the telephone survey mode. A previous Norwegian study found that postal nonresponders were more likely to be smokers but had fewer symptoms (23). Telephone responders are probably less symptomatic and hence probably perceive little benefit from participating in a postal questionnaire. When questioned directly on the telephone, resistance to response may be overcome. Our findings support this theory.
For differences in prevalence, our study had substantial power. The tests of changes in risk factors with survey effort would have smaller power. There was also some concern about multiple statistical tests, especially for the interactions of risk factors with survey effort. However, we tested a limited number of plausible interactions in a hierarchic fashion designed to reduce the number of false positives (48). There may have been differences in exposure-disease associations for other symptoms, but our data could not provide enough precision to pinpoint them. Differences for other risk factors may also have been possible. These we did not test, to avoid multiple comparisons.
The systematic literature search found 11 respiratory epidemiology articles that contained some data on nonresponse bias (22, 23, 3341). The response rates to the first postal questionnaire varied from less than 22 percent to 69 percent. A number of reasons for this variation are possible. Monso et al. (41) had a very low response rate, which, even after one reminder, was only 22 percent. These authors used a 1991 census sample frame, and it appears that the study was initiated in or about 1996. The study by Howard et al. (33) was conducted in a bilingual Hispanic population and used a commercial household directory that was not as up-to-date. Clearly, a prerequisite for a high response rate is a good-quality, up-to-date sampling frame. Even so, the remaining initial response rates varied from less than 41 percent to 69 percent. All the Nordic studies (23, 37, 38, 40) except ours had initial postal response rates of 60 percent or more. These studies were all based on mandatory universal population registries. Both Chinn et al. (34) and de Marco et al. (39) used a population-based register, but ones that were used for health care purposes, so quality may have been less than optimal.
All studies that reported on smoking found that smokers were late responders (table 4) (22, 23, 3739, 41). The relation of intensity of follow-up with symptoms and diagnoses was not universal; one study showed no relation (22), others found an increased prevalence of symptoms with intensity of follow-up (23, 37, 38), and others found a decreased prevalence of symptoms with follow-up (36, 39). None of these studies adjusted for potential confounders. However, follow-up had little effect on the final prevalence.
Only two studies are known to have investigated exposure-disease associations by survey effort (22, 23). One found a difference in an exposure-disease relation that caused occupational exposure to be no longer significant in a stepwise regression (23), but both reported little effect on final exposure-disease relations (22, 23).
The total sample size was moderate to large in most of these studies, but the number of respondents in follow-ups was relatively small (table 4). For example, the mean number of respondents in the telephone responder group was 389. If the prevalence of a symptom was 10 percent in a group with 389 respondents, the 95 percent confidence interval would be 7.2 percent to 13.4 percent. If this prevalence was compared with the prevalence in a larger group of initial responders, this confidence interval might obscure an important difference. In our study, the impact of follow-up on confidence intervals was small. Smaller studies may find it necessary to include a follow-up on the basis of study power alone. Most etiologic studies in respiratory epidemiology require adjustment for confounding, and doing so increases the required sample size. This adjustment is especially important for continuous variables for which sparseness of data is likely to be a problem.
Drane (31) proposed a linear or exponential extrapolation of trends in prevalence by reminder letter (32), which is attractive in its simplicity. This method has seen some use (34, 36, 39) but no thorough validation. There is little theoretical or empirical justification for selection of a linear or exponential trend. A what-if scenario with plausible alternatives might be just as reasonable. One plausible alternative is that the latest responders are most "equal" to the nonresponders and that prevalences for the nonresponders should be "set" to the prevalence of these latest responders (5). We did not explore this alternative, since neither of these techniques is sophisticated enough to cater to the multivariate situation, for example, when exposure-disease estimates are estimated and are controlled for confounding (49).
A cautious conclusion is that, so far, no studies have found important nonresponse bias in respiratory epidemiology health surveys that used this method, in either prevalence estimates or exposure-disease associations. A number of studies in other fields of epidemiology have used the same "wave" or willingness-to-respond approach (6, 820). These studies cover a wide range of geographic areas, different populations, and a variety of subjects. Most have found trends in the prevalence of outcomes or risk factors by willingness to respond, but the effect on final estimates has been variable. Some have found strong differences (7, 18, 19), whereas most others have found only small differences (6, 817, 20, 22, 23). The effect on exposure-disease associations (e.g., odds ratios) has rarely been studied (20).
Surprisingly little is known regarding the consistent refusers, the residual nonresponders, which probably reflects the difficulty and cost involved in obtaining information on these persons. In our study, we had no information on the residual 20 percent of nonresponders, except for age, gender, and geographic location. In an extreme scenario, both prevalences and associations with risk factors could be quite different on a population level. If the trends observed for the telephone responder group persisted or increased, results on a population level might be somewhat different. Our own literature search (5055) and various PubMed searches (National Library of Medicine, Bethesda, Maryland) revealed a single study (12) with a very high response rate (e.g., >90 percent). In a later commentary, Sheikh took a skeptical view of nonresponse (56), although he offered few solid references.
A limited amount of data is available from some studies that have compared nonresponders and responders regarding information external to the study. This use of external information to track nonresponse bias has shown that nonresponders are likely to under- and overreport previous illnesses (57, 58), have higher admission rates to the hospital (20, 59), and are more likely to have a history of psychiatric illness (60). They may also have higher mortality (20, 6164) and differences in health care utilization (65)possibly related to smoking (66). A number of mostly nonmedical studies are discussed by Wentland and Smith (51).
There is potentially a large nonresponse bias in most epidemiologic studies, or indeed in any survey. Outcomes are usually uncommon (e.g., <10 percent prevalence), and response rates are rarely above 90 percent. With a 90 percent response rate, a 5 percent prevalence among the responders may reflect a true prevalence of 4.514.5 percent in the whole population, given that the nonresponders have either a zero or a 100 percent prevalence of the outcome. This scenario may seem extreme but is probably the reason that response rates receive much attention, even given the fact that good examples of clear nonresponse bias are not abundant. In fact, nonresponse rates are sometimes used as the only indicator to judge the quality of a survey, which may be an oversimplification (67).
There are two possible interpretations of the present study. On the one hand, one may choose to use a larger sample size up front and a less-intense follow-up survey effort if it has already been established that nonresponse bias is not likely to be a large problem in the same population with the same survey topic. It would save time and money to use a large sample size and eliminate telephone follow-up to achieve a 5060 percent response rate rather than to select a smaller sample size and aim for a response rate of 7080 percent with follow-up. The larger survey would achieve about the same total number of responses and hence study power but in a shorter time and at a lower cost. In particular, the cellular telephone survey netted very few additional responses and required a substantial amount of time and work.
On the other hand, there is a large potential for nonresponse bias in all surveys, as noted above. A study may be greeted with skepticism if its response rate is as low as 43 percent, as it was in the present study. However, our study does show that pushing response rates from 60 percent to 80 percent would have been unlikely to materially affect conclusions since the additional reluctant responders were not very different from early responders. This finding is supported to some extent in smaller studies on other survey topics (6, 8, 9, 13, 15). If this finding can be replicated in other populations and for other survey topics, it could represent important savings of money and manpower.
In this study, the additional contribution of sending reminder letters and conducting a telephone follow-up to prevalence estimates and exposure-disease relations in respiratory epidemiology was small. In this population, a single letter would have been enough. The possibility that this finding can be generalized to other populations should be studied.
![]() |
ACKNOWLEDGMENTS |
---|
The assistance of the Interview Office of Statistics Norway is acknowledged. Karl Mørkved conceived of the idea of using cellular telephones.
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|