ESTIMATING NON-RESPONSE BIAS IN A SURVEY ON ALCOHOL CONSUMPTION: COMPARISON OF RESPONSE WAVES

Viviënne M. H. C. J. Lahaut1,*, Harrie A. M. Jansen1, Dike van de Mheen1, Henk F. L. Garretsen1,2, Jacqueline E. E. Verdurmen3 and Ad van Dijk4

1 Addiction Research Institute (IVO), Heemraadssingel 194, 3021 DM Rotterdam,
2 Tilburg University, PO Box 90153, 5000 LE Tilburg,
3 Trimbos Institute, Netherlands Institute of Mental Health and Addiction, PO Box 725, 3500 Utrecht and
4 Rotterdam Municipal Health Service, PO Box 70032, 3000 LP Rotterdam, The Netherlands

Received 23 May 2002; in revised form 11 September 2002; accepted 30 September 2002


    ABSTRACT
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Aims: According to ‘the continuum of resistance model’ late respondents can be used as a proxy for non-respondents in estimating non-response bias. In the present study, the validity of this model was explored and tested in three surveys on alcohol consumption. Methods: The three studies collected their data by means of mailed questionnaires on alcohol consumption whereby two studies also performed a non-response follow-up. Results: Comparisons of early respondents, late respondents and non-respondents in one study showed some support for ‘the continuum of resistance model’, although another study could not confirm this result. Comparison of alcohol consumption between three time response groups showed no significant linear pattern of differences between response waves. Conclusions: The hypothesis that late respondents are more similar to non-respondents than early respondents, could not be confirmed or rejected. Repeated mailings are effective in obtaining a greater sample size, but seem ineffective in improving the representativeness of alcohol consumption surveys.


    INTRODUCTION
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Survey researchers are constantly confronted with the problem of non-response. Moreover, with a low response rate, the validity of survey results may well be questioned. However, a low response rate does not necessarily mean that the results are biased. Non-response bias occurs when respondents and non-respondents differ in outcome variable(s), in which case the population parameters of these variable(s) can be over- or under-estimated.

Methods are available for survey researchers to deal with the problem of non-response. One is to build in strategies during survey development and data collection, in order to positively influence the response rate. Such strategies include financial incentives, repeated mailings, and an appealing survey design. Dillman (2000)Go described these approaches in his so-called ‘Tailored Design Method’: a method to maximize both quantity and quality of responses. These approaches are sometimes successful, but biased estimates due to non-response may still remain. Non-response bias can be estimated and/or corrected for in various ways. An indirect approach is to weight cases, whereby weights are allocated to various substrata, which are mostly defined by background variables; this approach is justified if the background variables are strongly related to outcome variable(s). Direct approaches include collecting valid information from objective sources, or conducting a non-response follow-up to collect data on outcome variable(s) of non-respondents to get insight into differences between respondents and non-respondents. However, follow-up studies tend to be costly and time-consuming; moreover, it is often difficult to contact initial non-respondents and secure their further participation.

An alternative and widely used approach is to estimate the non-response bias by comparing early and late respondents; a late respondent is then used as a proxy for a non-respondent. The underlying assumption behind this approach is that every subject in the study population has a position on the response continuum that ranges from ‘will never respond’ to ‘will always respond’. Non-respondents will be concentrated on the side of ‘will never respond’. Subjects who require more reminders before they participate would have been non-respondents if the data collection had stopped earlier. Therefore, late respondents most resemble non-respondents. This assumption has been called ‘the continuum of resistance model’ (Lin and Schaeffer, 1995Go; Voogt et al., 1998Go).

Questions arise, however, about the validity of this ‘continuum of resistance model’. If it appears valid, then it is justified to use late respondents as proxies for non-respondents, and then repeated mailings will help lower the degree of possible non-response bias. However, Table 1Go shows that a recent literature review does not provide a consistent answer to these questions.


View this table:
[in this window]
[in a new window]
 
Table 1. Literature review
 
Some researchers found no support for the assumption that late respondents can be used as a proxy for non-respondents (Hébert et al., 1996Go; Larroque et al., 1999Go), whereas others found that non-response bias can be estimated by analysing late respondents (Dalecki et al., 1993Go; Lin and Schaeffer, 1995Go; Etter and Perneger, 1997Go; Ullman and Newcomb, 1998Go). In several surveys, refusal conversions or repeated mailings appeared to be useful in lowering the degree of biased estimates. Some researchers found no significant differences between different time groups of respondents, which suggests that the number of mailings or refusal conversions could be reduced without affecting the accuracy of survey estimates (Green, 1991Go; Dalecki et al., 1993Go; Hébert et al., 1996Go; Etter and Perneger, 1997Go; Trinkoff and Storr, 1997Go; Ullman and Newcomb, 1998Go; van Goor and Stuiver, 1998Go; Voogt et al., 1998Go; Larroque et al., 1999Go; Woodruff et al., 2000Go; Schmidt and Fletcher, 2001Go).

Only three studies in our literature review had main survey topics that addressed substance use (Trinkoff and Storr, 1997Go; Ullman and Newcomb, 1998Go; Woodruff et al., 2000Go). Trinkoff and Storr (1997)Go did not question the justification of using late respondents as non-respondents, but they did investigate differences in substance use rates by mailing. Ullman and Newcomb (1998)Go compared reluctant respondents with non-respondents, early respondents with reluctant respondents and with respondents who participated at different time intervals; to estimate substance use of non-respondents, they used data of earlier mailings in which these non-respondents had participated. Woodruff et al.(2000)Go also compared initial, reluctant and non-respondents, but focused more on the effects of incentives (such as financial incentives). All three studies drew their sample from specific subpopulations.

In a previous study, by means of a non-response follow-up (Lahaut et al., 2002Go), we investigated whether respondents and non-respondents differed in their alcohol consumption; the results showed a significantly higher abstention rate and also a higher proportion of frequent excessive drinkers among non-respondents.

In the present study, we first investigated whether ‘the continuum of resistance model’ fits the data of this follow-up study; this model is also tested in a data set from a larger Dutch survey on alcohol consumption. Then, we investigated whether repeated mailings are worthwhile to collect more representative data on our outcome variable, alcohol consumption. For this, two larger samples from the general population were used. These studies were conducted in the same time period and used similar questions about alcohol consumption. Specifically, we aimed to answer the following questions: (1) are late respondents more similar to non-respondents than early respondents?; and (2) are there differences in alcohol consumption between response waves?


    SUBJECTS AND METHODS
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Procedure and samples
For this study, three data sets were used. The data of all three studies were collected by means of mailed questionnaires on alcohol consumption. There were three mailing periods: (wave 1) the first mailing of the questionnaire accompanied by an explanatory letter; (wave 2) 2 weeks later a reminder was sent to those who had not yet responded; (wave 3) subjects who had not responded in the second wave, received another reminder with another copy of the same questionnaire. The questionnaire in each study used similar questions on alcohol consumption. Subjects, aged 16–69 years, were approached by mail in April/May 1999.

The sample of the first data set (called the small-scale Rotterdam survey) consisted of 310 persons, who were living in 25 postal areas in the centre of Rotterdam. This study performed a non-response follow-up study (wave 4) on a sample of 177 subjects, who were approached mainly by means of house visits without prior notice.

The random sample of the second study (called the Utrecht survey) consisted of 5229 persons drawn from the municipal registry. The study in Utrecht also included a non-response follow-up study (wave 4). A random sample, stratified for age, was drawn from all non-respondents and consisted of 662 persons who were approached by telephone.

The random sample of the third study (called the large-scale Rotterdam survey) was drawn from the municipal registry in Rotterdam and consisted of 3226 persons. The large-scale Rotterdam survey did not perform a non-response follow-up study.

Measures
The outcome variable in the three studies was alcohol consumption. Alcohol consumption was measured by four questions: (1) have you drunk any alcoholic beverages in the past year?; (2) how many units of alcoholic beverages do you drink on average in a typical week?; (3) please indicate for each day in the previous week how many units of alcoholic beverages you have drunk; and (4) have you ever drunk six or more units of alcoholic beverages on 1 day in the past 6 months?

Non-respondents of the small-scale Rotterdam survey and the Utrecht survey were asked several questions about alcohol consumption. In the two non-response follow-up surveys, the interviewer asked non-respondents whether they had drunk any alcoholic beverages in the past year. If the answer was yes, these participants were asked an additional question on alcohol consumption. In the follow-up of the small-scale Rotterdam survey, this question was: have you ever drunk six or more units of alcoholic beverages on 1 day in the past 6 months? The follow-up of the Utrecht survey used the question: please indicate for each day in the previous week how many units of alcoholic beverages you consumed on each day.

We constructed several variables that provided information on the amount of alcohol consumed. The variable ‘drinking alcohol’ made a distinction between abstainers and drinkers. According to the total alcohol intake in a typical week, the frequencies of drinkers were categorized as: 1–14 units/week, 15–28 units/week and >=29 units/week. This variable was called ‘total alcohol consumption in a typical week’. Based on the weekly recall question, the frequencies of the total alcohol intake consumed in the previous week were calculated (variable ‘total alcohol consumption in previous week’). The question whether the subjects had ever consumed six or more units of alcoholic beverages on one occasion and with what frequency (never; 1–5 times/half year; 1–3 times/month; 1–2 times/week; >=3 times/week) was used for constructing the variable ‘frequency of excessive drinking’.

Analysis
Differences in the distribution of alcohol consumption between response waves were analysed by cross-tabulation. Statistical significance was estimated by chi-squared tests. A P value <=0.05 was considered significant.


    RESULTS
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Response rates
The total gross response rate of the small-scale Rotterdam survey was 42.9% (n = 133). The first response wave yielded a response rate of 19.4%, the second response wave an additional 7.4% and the third response wave yielded an additional response of 16.1%. Of the 177 non-respondents in the small-scale Rotterdam’s non-response follow-up study, 22 persons had moved or were absent for a longer period. Finally, we reached 129 non-respondents, of whom 102 answered at least one question on alcohol consumption. The contact rate of the follow-up was 83.2% (129/155).

The survey in Utrecht had a total gross response rate of 55.5% (n = 2902). The response percentages of the three waves were 32.9, 10.7 and 11.6%, respectively. The response time of 19 persons was unknown (response percentage of 0.3%). In the non-response follow-up study of Utrecht, 370 correct telephone numbers of 662 selected non-respondents were found. The researchers contacted 254 non-respondents. During these contact attempts, 133 persons answered at least one question on alcohol. The contact rate of the follow-up was 38.4% (254/662).

The total gross response rate of the large-scale Rotterdam survey was 50.5% (n = 1630). The first wave yielded an additional 29.0%, the second wave an additional 9.5%, and the third response wave yielded an additional response of 12.1%.

Differences between early respondents, late respondents and non-respondents
Table 2Go gives data on differences in alcohol consumption between early (first wave) respondents, late (third wave) respondents and non-respondents (fourth wave respondents) from the small scale Rotterdam survey. A significantly higher proportion of abstainers was found among non-respondents, than among early respondents. There were no significant differences between early respondents and non-respondents in frequencies of excessive drinking. Comparison of alcohol consumption between late respondents and non-respondents showed no significant differences, except in excessive drinking with the frequency of 1–5 times/half year. There was a higher proportion of excessive drinkers with a frequency of 1–5 times/half year among late respondents, than among non-respondents. The proportion of abstainers was also higher among non-respondents than among late respondents (third wave respondents), although the difference between these two groups was not significant.


View this table:
[in this window]
[in a new window]
 
Table 2. Comparison of alcohol consumption between early respondents, late respondents and non-respondents in the small-scale Rotterdam survey
 
Table 3Go gives data on alcohol consumption of early and late respondents from the Utrecht survey, compared with non-respondents. These two comparisons showed no significant differences. Comparison of abstention rates between early respondents and non-respondents showed an almost significant difference (P value 0.057). The abstention rate in the Utrecht survey was higher among non-respondents, than among early respondents.


View this table:
[in this window]
[in a new window]
 
Table 3. Comparison of alcohol consumption between early respondents versus non-respondents and late respondents versus non-respondents in the Utrecht survey
 
Differences in alcohol consumption between the three response waves
Table 4Go gives data on the comparison of alcohol consumption between the first, second and third waves of respondents in the large-scale survey in Rotterdam and in the Utrecht survey. In both surveys, there were no significant differences between the three response waves, except for the proportion of abstainers. The proportion of abstainers in the Utrecht survey was the highest among the second wave respondents and the lowest proportion was found in the first response wave. This was not the case in the Rotterdam survey.


View this table:
[in this window]
[in a new window]
 
Table 4. Differences in alcohol consumption by response waves in the large-scale Rotterdam survey and in the Utrecht survey
 

    DISCUSSION
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
In this study, we aimed to answer the questions: (1) are late respondents more similar to non-respondents, than to early respondents?; and (2) are there differences in alcohol consumption between response waves? For the first question, we can neither confirm nor reject the hypothesis that late respondents are more similar to non-respondents, than to early respondents. In the small-scale Rotterdam survey, the abstention rate of late respondents was more similar to non-respondents, than to early respondents. Thus, this finding supports ‘the continuum of resistance model’. However, in the Utrecht survey, this result was not confirmed. For the second research question, neither of the surveys (i.e. the large-scale Rotterdam survey and Utrecht survey) showed a significant linear pattern of differences in alcohol consumption between response waves. We could not confirm the ‘continuum of resistance model’. The results suggest that repeated mailings are not necessary in order to collect more representative alcohol consumption estimates. The results do indicate, however, that repeated mailings are effective in obtaining a greater sample size. The two reminders in the small-scale Rotterdam survey were responsible for 23.5% of the total response rate (42.9%), compared with 22.3% of the total response rate (55.5%) in Utrecht. The reminders in the large-scale Rotterdam survey contributed an additional 21.6% to the total response rate of 50.5%. Both surveys, especially the mailing in the third wave, yielded a relatively high additional percentage to the total response rate.

Some limitations in our study have to be acknowledged. One limitation is the mode of data collection. The mode of the initial surveys (i.e. the mailed questionnaire) differed from that used in the follow-up of non-respondents (i.e. face-to-face interviews/telephone interviews). Additionally, both non-response follow-ups (the small-scale Rotterdam survey and the Utrecht survey) used different modes: i.e. mainly faceto-face interviews versus telephone interviews. Because of conflicting reports in the literature, it is difficult to assess the impact of the different methods used on self-reported alcohol consumption, and on responsiveness of non-respondents in the follow-up (Rehm and Arminger, 1996Go; Kraus and Augustin, 2001Go). There was a difference in contact rates between both our follow-up studies (83.2 vs 38.4%). Possible explanations for this difference may be the different modes used for each non-response follow-up and/or the number of contact attempts.

Besides the problem of using different modes, another limitation is the analysis itself. Most non-response follow-up studies suffer from low response rates. This is also true in our case, especially with regard to the Utrecht non-response follow-up. The response rates for questions on alcohol in the non-response follow-up study in Rotterdam and Utrecht were 79.1% (102/129) and 52.4% (133/254), respectively. This means that conclusions as to whether late respondents are more similar to non-respondents than to early respondents, rely on the assumption that the remaining non-respondents have the same pattern of alcohol consumption as fourth wave respondents. Owing to a low response rate, the results of the Utrecht non-response follow-up strongly relies on this assumption; however, the group of remaining non-respondents may become an increasingly deviant category. Although not in the field of substance use, the study by van Goor and Stuiver (1998)Go showed that the group of hard-core non-respondents differed more and more from the other response groups.

We were able to compare our results with three other studies in the field of alcohol and drugs. Our results showed no significant differences in alcohol consumption between first, second and third wave respondents. Similarly, Trinkoff and Storr (1997)Go found no significant differences in substance use rates by mailing. In the study of Ullman and Newcomb (1998)Go, the pattern of results was complex and indicated few differences in behavioural low social conformity variables (i.e. frequency of use of alcohol, cigarettes and marijuana) between early and reluctant respondents and between different groups of reluctant respondents. The results of Woodruff et al.(2000)Go showed significant differences in baseline smoking between ‘on time’ and ‘reluctant’ respondents. Smoking tobacco and marijuana use are similar to alcohol-sensitive survey topics. Ullman and Newcomb (1998)Go and Woodruff et al. (2000)Go included financial incentives in their reminders, which meant that they did not measure the relationship between response and substance use directly. Our results also suggest that there are no significant differences in alcohol consumption between late respondents and non-respondents, although the group differences in the Rotterdam survey were not small. Reluctant respondents and non-respondents in the study of Woodruff et al. (2000)Go showed considerable similarity in their baseline smoking characteristics.

The answer as to whether the alcohol consumption distribution of late respondents can be used as a proxy for the distribution of non-respondents remains ambiguous; our results are mixed. Thus, more studies are needed to establish whether late respondents can be used as proxies for non-respondents. Also, our results show no linear patterns of differences in alcohol consumption between response waves. The question arises as to whether it may be more economical to draw a larger initial sample, rather than sending reminders in order to enhance the response rate.

To estimate non-response bias, it is advisable to do an intensive non-response follow-up on a small representative sample of non-respondents, which allows more time for each individual subject, rather than an extensive follow-up on a large sample of non-respondents.


    FOOTNOTES
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
* Author to whom correspondence should be addressed. Back


    REFERENCES
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Dalecki, M. G., Whitehead, J. C. and Blomquist, G. C. (1993) Sample non-response bias and aggregate benefits in contingent valuation: an examination of early, late and non-respondents. Journal of Environmental Management 38, 133–143.[CrossRef][ISI]

Dillman, D. A. (2000) Mail and Internet Surveys: the Tailored Design Method. Wiley, New York.

Etter, J.-F. and Perneger, T. V. (1997) Analysis of non-response bias in a mailed health survey. Journal of Clinical Epidemiology 50, 1123–1128.[CrossRef][ISI][Medline]

Green, K. E. (1991) Reluctant respondents: differences between early, late, and nonresponders to a mail survey. Journal of Experimental Education 59, 268–276.[ISI]

Hébert, R., Bravo, G., Korner-Bitensky, N. and Voyer, L. (1996) Refusal and information bias associated with postal questionnaires and face-to-face interviews in very elderly subjects. Journal of Clinical Epidemiology 49, 373–381.[CrossRef][ISI][Medline]

Kellerman, S. E. and Herold, J. (2001) Physician response to surveys: a review of the literature. American Journal of Preventive Medicine 20, 61–67.[Medline]

Kraus, L. and Augustin, R. (2001) Measuring alcohol consumption and alcohol-related problems: comparison of responses from self-administered questionnaires and telephone interviews. Addiction 96, 459–471.[CrossRef][ISI][Medline]

Lahaut, V. M. H. C. J., Jansen, H. A. M., van de Mheen, D. and Garretsen, H. F. L. (2002) Non-response bias in a sample survey on alcohol consumption. Alcohol and Alcoholism 37, 256–260.[Abstract/Free Full Text]

Larroque, B., Kaminski, M., Bouvier-Colle, M.-H. and Hollebecque, V. (1999) Participation in a mail survey: role of repeated mailings and characteristics of nonrespondents among recent mothers. Paediatric and Perinatal Epidemiology 13, 218–233.[CrossRef][ISI][Medline]

Lin, I. F. and Schaeffer, N. C. (1995) Using survey participants to estimate the impact of nonparticipation. Public Opinion Quarterly 59, 236–258.[Abstract]

Rehm, J. and Arminger, G. (1996) Alcohol consumption in Switzerland 1987–93: adjusting for differential effects of assessment techniques on the analysis of trends. Addiction 91, 1335–1344.[CrossRef][ISI][Medline]

Schmidt, D. and Fletcher, J. (2001) Measuring response bias in survey research: an analysis of age characteristics of early respondents and resistors, pp. 1–13. California State University, Survey Research Center, Chico.

Trinkoff, A. M. and Storr, C. L. (1997) Collecting substance use data with an anonymous mailed survey. Drug and Alcohol Dependence 48, 1–8.[CrossRef][ISI][Medline]

Ullman, J. B. and Newcomb, M. D. (1998) Eager, reluctant, and nonresponders to a mailed longitudinal survey: attitudinal and substance use characteristics differentiate respondents. Journal of Applied Social Psychology 28, 357–375.[ISI]

van Goor, H. and Stuiver, B. (1998) A wave-analysis of distributional bias, substantive bias and data quality in a mail survey among Dutch municipalities. Acta Politica 33, 179–196.

Voogt, R. J. J., Saris, W. E. and Niemöller, B. (1998) Non-response, and the gulf between the public and politicians. Acta Politica 33, 250–280.

Woodruff, S. I., Conway, T. L. and Edwards, C. C. (2000) Increasing response rates to a smoking survey for U.S. Navy enlisted women. Evaluation and the Health Professions 23, 172–181.[Abstract/Free Full Text]





This Article
Abstract
FREE Full Text (PDF)
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Request Permissions
Google Scholar
Articles by Lahaut, V. M. H. C. J.
Articles by van Dijk, A.
PubMed
PubMed Citation
Articles by Lahaut, V. M. H. C. J.
Articles by van Dijk, A.