NON-RESPONSE BIAS IN A SAMPLE SURVEY ON ALCOHOL CONSUMPTION

Viviënne M. H. C. J. Lahaut,*, Harrie A. M. Jansen, Dike van de Mheen and Henk F. L. Garretsen

Addiction Research Institute (IVO), Heemraadssingel 194, 3021 DM Rotterdam, The Netherlands

Received 11 July 2001; in revised form 30 October 2001; accepted 12 November 2001


    ABSTRACT
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
— In a non-response follow-up study, non-respondents of the original mailed questionnaire were approached again by house visits in order to compare their alcohol consumption with that of the respondents of the same mailed questionnaire. Differences in alcohol consumption between respondents and non-respondents were found. There is strong evidence for overrepresentation of non-response among abstainers, but weak evidence among frequent excessive drinkers.


    INTRODUCTION
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
During recent decades, survey response rates in most Western countries have declined. In The Netherlands, response rates of official governmental surveys are very low, with rates dropping from 80% in the 1980s to ~60% at the end of the 1990s; moreover, this negative trend is continuing. The low response rates in The Netherlands may be due partly to the fact that participation in governmental surveys is not mandatory (De Heer, 1999Go).

Non-response leads to a smaller final sample size and, therefore, to a loss of accuracy in population estimates. However, if the non-response is not related to the research variable of interest, taking larger samples can compensate for this loss. Conversely, if non-response is directly related to the research topic, errors may occur, which can seriously distort the survey results. This non-response bias occurs when a significant number of people in the survey sample fail to respond and have relevant characteristics that differ from those who do respond (Dillman, 2000Go). In such cases, the non-response is selective.

A widely used method to correct for a non-response bias is corrective weighting of the survey data by use of socio-demographic variables (Lemmens et al., 1988Go; Bongers et al., 1997Go). However, it has been shown that this method does not correct the bias sufficiently, because the inherent assumption that respondents and non-respondents within the same socio-demographic category are also equal on the outcome variable seems untenable (Bradburn, 1992Go; Presser and Traugott, 1992Go; Van Goor and Stuiver, 1995Go).

In the field of alcohol research, non-response is frequently considered as a causal factor of the discrepancy between the survey-based estimates and the official (taxes-) based estimates of alcohol consumption. Pernanen (1974) has reported that many survey subpopulations with a higher proportion of heavy drinkers tend to show higher non-response rates; this can cause underreporting of the survey data on alcohol consumption, compared to the official sales data on alcohol consumption. Garretsen (1983) showed an underreporting of both the frequency and the quantity of alcohol consumption. The underestimation varied according to the subpopulations; for example, underreporting of alcohol consumption was higher among women than men, but there was no support for a higher non-response among the problem drinkers. Knibbe (1982) found some evidence for a higher percentage of abstainers and ‘excessive’ drinkers among the non-respondents whereas Lemmens et al. (1988) found no support for the hypothesis of higher alcohol consumption among non-respondents, compared with respondents. In the same study, Lemmens and colleagues did show that female non-respondents generally drank less and had higher abstention rates. Furthermore, although occasional heavy alcohol use in the previous 6 months was more frequent among male non-respondents than among male respondents, the reverse was true for frequent heavy alcohol use. A follow-up of non-respondents of the (American) National Household Survey of Drug Abuse showed no differences between non-respondents and respondents in the prevalence of alcohol consumers (Caspar, 1992Go). Gmel (2000) found no differences between non-respondents and respondents in higher alcohol consumption; although the mean consumption and the percentage of heavy drinkers were higher in non-respondents, these differences were not significant. Wild et al. (2001) showed that non-responders to a follow-up questionnaire on alcohol use and beliefs about drinking reported consuming five or more standard drinks once per week or more at a slightly greater rate (19.1%) than did respondents (14.3%).

A limitation of most follow-up studies among non-respondents is the relatively low response rates of these secondary surveys. In the follow-up of non-respondents, generally there are many initial non-respondents who cannot be reached or who refuse again (Sosdian and Sharp, 1980Go; Jansen and Hak, 1999Go). It is often assumed that excessive drinkers and/or problem drinkers are difficult to reach because of their lifestyle characteristics (e.g. homelessness, seldom at home, not answering their mail). It is also assumed that excessive drinkers and/or problem drinkers, if reached, are more likely to refuse to participate in a survey than light and moderate drinkers.

Various models and theories have been developed to identify and explain non-response. Dillman (2000) applied the social exchange theory as a basis for survey design methods aiming at maximizing the response rate. According to this theory, actions of individuals, in this case responding or not responding, can be predicted on the basis of three elements: rewards, costs and trust. Rewards are what one expects to gain from a particular activity, e.g. social validation, appreciation, ‘liking to dO', or tangible rewards, such as money. In conducting a survey, a phrase such as ‘we very much appreciate your help’ has a reward value for many people. ‘Liking to dO' is also a powerful determinant of behaviour; most people enjoy participating in a survey if the topic is of interest to them. Costs are what one gives up or spends to obtain the rewards, e.g. inconvenience, embarrassment, feeling subordinated, anxiety, great physical or mental effort, or invasion of privacy. A simple act such as inclusion of a (paid) envelope for a mailed questionnaire increases the response rate, whereas questions that may cause embarrassment or anxiety lower the response rate. The third element in the social exchange theory is trust. This is the expectation that in the long run the rewards of doing something will outweigh the costs, e.g. participants may see a legitimate authority or a token of appreciation in advance as a form of trust.

On a more concrete level, Groves et al. (1992) reported various (reward/cost) factors which can influence survey participation. They distinguished factors at four levels: i.e. societal-level factors; attributes of the survey design; attributes of the interviewer and respondent–interviewer interaction; and characteristics of the sample person. Examples of societal-level factors are the survey-taking climate, the legitimacy of societal institutions or social cohesion. Attributes of the survey design can be the mode of the survey, the length of the questionnaire or the survey topic. Relevant personal characteristics are age, gender, income and health status of the sample person. These latter characteristics are also important for the interviewer. Factors related to respondent–interviewer interaction can include strategies of the interviewer to persuade the respondents or expectations of the interviewer and the interviewee.

These theories are tools for understanding the decision to participate (or not) in a survey, but do not specifically address the question of non-response bias. Most factors mentioned by Groves et al. (1992) may cause bias only in an indirect way, whereas non-response bias is strongest when respondents select themselves in relation to the perceived topic of the survey. Therefore, here the focus is on ‘topic’ as the main explanatory factor of non-response in our earlier survey on alcohol consumption. The two main aspects of the topic with regard to responsiveness are reported to be salience and social desirability. Heberlein and Baumgartner (1978) showed that salience (interested or not interested in the topic) has a strong influence on the response rate. When the topic of the survey is salient to the respondent, the costs of responding may be reduced. Martin (1994) verified that people's interest in the survey topic can have considerable impact on response rates: persons were almost twice as likely to participate if the topic was of high interest. Similarly, Dillman and Carley-Baxter (2000) showed that salience is a significant determinant of response rate. Others have shown that people who feel threatened by a topic behave socially less desirably or feel embarrassed/ashamed with respect to the survey topic and thus respond less (Gannon et al., 1971Go; Green, 1991Go). For example, socially acceptable behaviour, such as exercise and good nutrition, may be frequently overreported, whereas undesirable behaviour, such as smoking and drinking, may be underreported (Warnecke et al., 1997Go; Bongers, 1998Go). Van Goor and Stuiver (1995) showed a pattern of overrepresentation of non-response in both ‘extreme’ categories of his outcome variable, i.e. the effectiveness of governmental organizations in policy making with regard to trailer camps. In his explanation of this pattern, the factor ‘topic’ plays an important role in that the very effective organizations lost interest in the topic of effectiveness and the less effective organizations did not respond because they felt threatened by the survey because of their poor performance. This latter study gave rise to a two-tailed pattern of non-response bias. We hypothesize that this pattern can also be applied to surveys on alcohol consumption. Two hypotheses are formulated: (1) non-response is high at the lowest end of the research variable, in our case the abstainers; (2) non-response is also high at the upper end of the research variable, in our case the frequent excessive drinkers. These hypotheses rely on the same reasoning as used by Van Goor and Stuiver (1995). Abstainers do not respond because they have no experience with alcohol and therefore may not be interested in it and/or may not see the relevance of their response. Frequent excessive drinkers do not respond because this group may perceive their amount of consumption as socially undesirable.

In the present study, the two hypotheses were tested by conducting a follow-up investigation of non-respondents to a mailed questionnaire on alcohol consumption, in order to compare them with respondents.


    SUBJECTS AND METHODS
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Procedure and sample
The non-response follow-up study presented here is part of a main methodological study conducted in 1999 on alcohol surveys. For the main study, a random sample of 1000 persons aged 16–69 years was drawn from the municipal registry in Rotterdam, The Netherlands. These people received a mailed questionnaire on alcohol consumption. After two reminders (one of which included the questionnaire again), the overall response rate was 44%. In order to facilitate and intensify our follow-up study on non-response, we selected 25 postal areas in the centre of Rotterdam. This sample (the ‘secondary’ sample) consisted of 310 persons: 133 respondents of the primary survey (i.e. the mailed questionnaire) and 177 non-respondents. These persons were designated as primary respondents and primary non-respondents, respectively. The persons who did or did not respond to our non-response survey were called secondary respondents and secondary non-respondents, respectively.

Information about the primary respondents' alcohol use was derived from their answers to two alcohol questions in the mailed questionnaire, which were completed by all of them. All primary non-respondents were approached by means of house visits without prior notice. A maximum of five attempts was made to reach the right person at home. During the fieldwork, 22 of the 177 primary non-respondents appeared to have moved house or were absent for a long period; these persons were deemed ineligible. Moreover, it proved impossible to contact another 26 primary non-respondents (they were not at home or did not open the door). Finally, having lost 48/177 persons to follow-up, 129 primary non-respondents were contacted during the house visits; this yielded a contact rate of 83% [129/(177 – 22)]. Of the several questions asked of the primary non-respondents, two were about alcohol use. Of the 129 primary non-respondents, 102 answered at least one alcohol question and 80 of these 102 secondary respondents answered both alcohol questions; this yielded a net response rate of 52% [80/(177 – 22)].

Both the primary respondents and secondary respondents were asked the same two alcohol questions.

Measures
Alcohol consumption was measured by the following two items: ‘Do you ever drink alcoholic beverages?’ and ‘Do you ever drink six or more units of alcoholic beverages in one day?’ The frequency of the alcohol consumption was assessed in the first instance by a 9-point scale (every day, more than 3 times a week, 2 or 3 times a week, once a week, 2 or 3 times a month, once a month, less than once a month, never drinking >=6 units in one day and never drinking alcohol at all). Table 1Go presents the classification used for alcohol consumption. This classification is based on a combination of those used by Garretsen (1983) and Wild (2001). The socio-demographic measures used for the analyses included: gender, age and nationality. Age was classified as a categorical variable (16–25, 26–35, 36–69 years). Nationality was divided into two categories: Dutch only and non-Dutch only.


View this table:
[in this window]
[in a new window]
 
Table 1. Classification of alcohol consumption
 
Statistical analyses
The differences in self-reported alcohol consumption between the primary respondents and secondary respondents (i.e. primary non-respondents who responded to our non-response survey) were first analysed by bivariate cross-tabulation. Next, the cross-tabulation was stratified by gender, age and nationality. Statistical significance was estimated by the {chi}2-test. Each category of drinking frequency was tested separately. P <= 0.05 was regarded as statistically significant.


    RESULTS
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Table 2Go gives information on the differences between primary and secondary respondents in their self-reported alcohol consumption. There were higher proportions of frequent excessive drinkers and of abstainers among secondary respondents than among primary respondents. The proportions of occasional excessive and moderate drinkers were greater among primary respondents than among secondary respondents. Thus, non-response is highest among the frequent excessive drinkers and abstainers. When stratified by gender (Table 3Go), a trend to overrepresentation of non-response of both abstainers and frequent excessive drinkers was seen amongst males and amongst females, but the differences for males were not significant.


View this table:
[in this window]
[in a new window]
 
Table 2. Distribution of alcohol consumption of primary respondents and secondary respondents (%)
 

View this table:
[in this window]
[in a new window]
 
Table 3. Distribution of alcohol consumption of primary respondents and secondary respondents by gender (%)
 
Table 4Go shows that, in the age group of 16–25 years, there were relatively more abstainers among the secondary respondents, whereas there were relatively more frequent excessive drinkers among the primary respondents. An overrepresentation of non-response on both ‘extreme’ sides of the variable ‘alcohol consumption’ was seen in the age group 26–35 years. Relatively more secondary respondents aged 36–69 years were frequent excessive drinkers, occasional excessive drinkers and abstainers. Significant differences in alcohol consumption between primary and secondary respondents were found for moderate drinkers aged 26–35 and 36–69 years and for abstainers aged 16–25 and 26–35 years. Table 5Go gives data on the differences between primary and secondary respondents in their self-reported alcohol consumption after stratification by nationality. Dutch secondary respondents tended to be abstainers more often than the Dutch primary respondents, and were less likely to be moderate drinkers. Among the non-Dutch, the abstention rate was also higher than among the primary respondents. (No frequent excessive drinkers were found among either the non-Dutch primary respondents or the non-Dutch secondary respondents.)


View this table:
[in this window]
[in a new window]
 
Table 4. Distribution of alcohol consumption of primary respondents and secondary respondents by age (%)
 

View this table:
[in this window]
[in a new window]
 
Table 5. Distribution of alcohol consumption of primary respondents and secondary respondents by nationality (%)
 
To summarize: this study showed significant underrepresentation of several categories in the primary survey: (1) abstainers were underrepresented, whereas frequent excessive drinkers were not significantly underrepresented; (2) the underrepresentation of abstainers is greater for females than for males, greater for those aged <35 years, and is greater for the Dutch than for the non-Dutch group.


    DISCUSSION
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
This study first explored whether there were any differences in alcohol consumption between primary respondents (i.e. respondents in our primary survey, the mailed questionnaire) and secondary respondents (i.e. primary non-respondents who responded only to our secondary survey, the non-response follow-up study). Then, the hypothesis was tested that non-response is overrepresented in the extreme categories of our alcohol consumption variable. There was strong evidence for overrepresentation of non-response abstainers. The differences in the abstention rates between the primary and secondary respondents were also present when data were stratified by gender, age and nationality. However, the evidence for differences in the other extreme, i.e. the frequent excessive drinkers, was weak and inconsistent. This latter result may be due to the low prevalence of frequent excessive drinking in the study population, which led to very small numbers in the relevant categories.

Some other limitations of the study should be addressed. First, because the secondary non-response rate was ~52%, our conclusions rely on the assumption that the secondary non-response has the same effects on the estimates of alcohol consumption as the primary non-response. Another limitation concerns the representativeness of the sample. Only 25 postal areas in the city centre were included to obtain the sample for the follow-up study and the age group 36–69 years was underrepresented in this second sample. Nevertheless, there were no significant differences in gender and nationality between the primary survey sample and the secondary survey sample. The study methods also differed. In the primary survey, a mailed questionnaire was used, whereas the follow-up of non-respondents consisted of face-to-face interviews. It has been suggested that, due to greater anonymity, mail surveys encourage fuller report of sensitive topics than face-to-face interviews (De Leeuw, 1992Go; Gmel, 2000Go; Kraus and Augustin, 2001Go), whereas others found higher reported alcohol consumption in personal interviews (Cutler et al., 1988Go; Rehm and Arminger, 1996Go). Because of these conflicting reports, it is difficult to assess the impact of the different methods used in our alcohol surveys.

Despite these limitations, the results of this study reveal a serious non-response bias in our primary survey, which is directly related to the topic of the survey, namely alcohol consumption. This bias cannot be corrected by weighting data on the basis of socio-demographic variables, because, within our subgroups, the same bias exists. Therefore, the aim should be to minimize non-response in a survey by, for example, developing more appealing survey materials. Furthermore, this study confirms the need for a thorough non-response follow-up study to evaluate non-response biases, as also emphasized by Hill and Roberts (1997) in the field of lifestyle surveys. Because of the time-consuming effort required to contact primary non-respondents, it is advisable to restrict the follow-up study to a small subsample, thus allowing more attention to be given to the individual subject.


    FOOTNOTES
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
* Author to whom correspondence should be addressed. Back


    REFERENCES
 TOP
 FOOTNOTES
 ABSTRACT
 INTRODUCTION
 SUBJECTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Bongers, I. B. M. (1998) Problem drinking among the general population: a public health issue? Dissertation, Addiction Research Institute (IVO), Rotterdam.

Bongers, I. M. B., Lemmens P. H. H. M., Oers, H. A. M. van and Tan, F. E. S. (1997) Methode ter correctie voor vertekening van surveyresultaten ten gevolge van indirecte selectieve non-respons. Tijdschrift voor Sociale Gezondheidszorg 75, 122–128.

Bradburn, N. M. (1992) A response to the nonresponse problem. Public Opinion Quarterly 56, 391–397.[ISI]

Caspar, R. (1992) Follow-up of non-respondents in 1990. In Survey Measurement of Drug Use, Turner, C. F., Lessler, J. T. and Gfroerer, J. C. eds, pp. 155–173. National Institute on Drug Abuse, Rockville.

Cutler, S. F., Wallace, P. G. and Haines, A. P. (1988) Assessing alcohol consumption in general practice patients — a comparison between questionnaire and interview. Alcohol 23, 441–450.

De Heer, W. (1999) International response trends: results of an international survey. Journal of Official Statistics 15, 129–142.

De Leeuw, E. D. (1992) Data quality in mail, telephone, and face-to-face surveys. Dissertation, Vrije Universiteit Amsterdam, Amsterdam.

Dillman, D. A. (2000) Introduction to tailored design. In Mail and Internet Surveys: The Tailored Design Method, Dillman, D. A. ed., pp. 3–31. Wiley, New York.

Dillman, D. A. and Carley-Baxter, L. R. (2000) Structural determinants of mail survey response rates over a 12 year period, 1988–1999. Proceedings of the section on survey method. The American Statistical Association, Alexandria, VA. Published on web page http://survey.sesrc.wsu.edu/dillman

Gannon, M. J., Northern, J. C. and Carrol, S. J., Jr (1971) Characteristics of nonrespondents among workers. Journal of Applied Psychology 55, 586–588.[ISI]

Garretsen, H. F. L. (1983) Probleemdrinken: prevalentiebepaling, beïnvloedende factoren en preventiemogelijkheden. Dissertation, Swets and Zeitlinger B.V., Lisse.

Gmel, G. (2000) The effect of mode of data collection and of non-response on reported alcohol consumption: a split-sample study in Switzerland. Addiction 95, 123–134.[ISI][Medline]

Green, K. E. (1991) Reluctant respondents: differences between early, late and nonresponders to a mail survey. Journal of Experimental Education 56, 268–276.

Groves, R. M., Cialdini, R. B. and Couper, M. P. (1992) Understanding the decision to participate in a survey. Public Opinion Quarterly 56, 475–495.[Abstract]

Heberlein, T. A. and Baumgartner, R. M. (1978) Factors affecting response rates to mailed questionnaires: a quantitative analysis of the published literature. American Sociological Review 43, 447–462.[ISI]

Hill, A. and Roberts, J. (1997) Non-response bias in a lifestyle survey. Journal of Public Health in Medicine 19, 203–207.

Jansen, H. and Hak, T. (1999) Nonresponse to mail surveys in a lower-class urban area — a two stage exploration of access failure and refusal. Bulletin de Méthodologie Sociologique 62, 5–27.

Knibbe, R. A. (1982) Probleemdrinken in Limburg. Rijkuniversiteit Limburg, Maastricht.

Kraus, L. and Augustin, R. (2001) Measuring alcohol consumption and alcohol-related problems: comparison of responses from self-administered and telephone interviews. Addiction 96, 459–471.[ISI][Medline]

Lemmens, P. H. H. M., Tan, E. S. and Knibbe, R. A. (1988) Bias due to non-response in a Dutch survey on alcohol consumption. British Journal of Addiction 83, 1069–1077.[ISI][Medline]

Martin, C. L. (1994) The impact of topic interest on mail survey response behaviour. Journal of Market Research Society 36, 327–338.

Pernanen, K. (1974) Validity of survey data on alcohol use. In Research Advances in Alcohol and Drug Problems, Vol. 1, Gibbens, R., Israël, Y., Kalant, H., Popham, R., Schmidt, W. and Smart, R. G. eds, pp. 355–374. Wiley, New York.

Presser, S. and Traugott, M. (1992) Little white lies and social science models: correlated response errors in a panel study of voting. Public Opinion Quarterly 56, 77–86.[Abstract]

Rehm, J. and Arminger, G. (1996) Alcohol consumption in Switzerland 1987–93: adjusting for differential effects of assessment techniques on the analysis of trends. Addiction 91, 1335–1344.[ISI][Medline]

Sosdian, C. P. and Sharp, L. M. (1980) Non-response in mail surveys: access failure or respondent resistance. Public Opinion Quarterly 44, 396–402.[Abstract]

Van Goor, H. and Stuiver, B. (1995) Succes en falen van beleid en non-respons: een empirisch onderzoek naar het terugzenden van schriftelijke enquêtes door Nederlandse gemeenten. Sociologische Gids 42, 388–406.

Warnecke, R. B., Johnson, T. P., Chávez, N., Sudman, S., O'Rourke, D. P., Lacey, L. and Horm, J. (1997) Improving question wording in surveys of culturally diverse populations. Annals of Epidemiology 7, 334–342.[ISI][Medline]

Wild, T. C. (2001) Effects of personal drinking on perceived norms for alcohol consumption, approval of others' drinking, and criteria for defining alcohol problems. Paper presented at the 27th Annual Alcohol Epidemiology Symposium of the Kettil Bruun Society for Social and Epidemiological Research on Alcohol, 28 May–1 June, 2001, Toronto, Canada.

Wild, T. C., Cunningham, J. and Adlaf, E. (2001) Nonresponse in a follow-up to a representative telephone survey of adult drinkers. Journal of Studies on Alcohol 62, 257–261.[ISI][Medline]