1 Addiction Research Institute (IVO), Heemraadssingel 194, 3021 DM Rotterdam,
2 Tilburg University, PO Box 90153, 5000 LE Tilburg,
3 Trimbos Institute, Netherlands Institute of Mental Health and Addiction, PO Box 725, 3500 Utrecht and
4 Rotterdam Municipal Health Service, PO Box 70032, 3000 LP Rotterdam, The Netherlands
Received 23 May 2002; in revised form 11 September 2002; accepted 30 September 2002
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods are available for survey researchers to deal with the problem of non-response. One is to build in strategies during survey development and data collection, in order to positively influence the response rate. Such strategies include financial incentives, repeated mailings, and an appealing survey design. Dillman (2000) described these approaches in his so-called Tailored Design Method: a method to maximize both quantity and quality of responses. These approaches are sometimes successful, but biased estimates due to non-response may still remain. Non-response bias can be estimated and/or corrected for in various ways. An indirect approach is to weight cases, whereby weights are allocated to various substrata, which are mostly defined by background variables; this approach is justified if the background variables are strongly related to outcome variable(s). Direct approaches include collecting valid information from objective sources, or conducting a non-response follow-up to collect data on outcome variable(s) of non-respondents to get insight into differences between respondents and non-respondents. However, follow-up studies tend to be costly and time-consuming; moreover, it is often difficult to contact initial non-respondents and secure their further participation.
An alternative and widely used approach is to estimate the non-response bias by comparing early and late respondents; a late respondent is then used as a proxy for a non-respondent. The underlying assumption behind this approach is that every subject in the study population has a position on the response continuum that ranges from will never respond to will always respond. Non-respondents will be concentrated on the side of will never respond. Subjects who require more reminders before they participate would have been non-respondents if the data collection had stopped earlier. Therefore, late respondents most resemble non-respondents. This assumption has been called the continuum of resistance model (Lin and Schaeffer, 1995; Voogt et al., 1998
).
Questions arise, however, about the validity of this continuum of resistance model. If it appears valid, then it is justified to use late respondents as proxies for non-respondents, and then repeated mailings will help lower the degree of possible non-response bias. However, Table 1 shows that a recent literature review does not provide a consistent answer to these questions.
|
Only three studies in our literature review had main survey topics that addressed substance use (Trinkoff and Storr, 1997; Ullman and Newcomb, 1998
; Woodruff et al., 2000
). Trinkoff and Storr (1997)
did not question the justification of using late respondents as non-respondents, but they did investigate differences in substance use rates by mailing. Ullman and Newcomb (1998)
compared reluctant respondents with non-respondents, early respondents with reluctant respondents and with respondents who participated at different time intervals; to estimate substance use of non-respondents, they used data of earlier mailings in which these non-respondents had participated. Woodruff et al.(2000)
also compared initial, reluctant and non-respondents, but focused more on the effects of incentives (such as financial incentives). All three studies drew their sample from specific subpopulations.
In a previous study, by means of a non-response follow-up (Lahaut et al., 2002), we investigated whether respondents and non-respondents differed in their alcohol consumption; the results showed a significantly higher abstention rate and also a higher proportion of frequent excessive drinkers among non-respondents.
In the present study, we first investigated whether the continuum of resistance model fits the data of this follow-up study; this model is also tested in a data set from a larger Dutch survey on alcohol consumption. Then, we investigated whether repeated mailings are worthwhile to collect more representative data on our outcome variable, alcohol consumption. For this, two larger samples from the general population were used. These studies were conducted in the same time period and used similar questions about alcohol consumption. Specifically, we aimed to answer the following questions: (1) are late respondents more similar to non-respondents than early respondents?; and (2) are there differences in alcohol consumption between response waves?
![]() |
SUBJECTS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The sample of the first data set (called the small-scale Rotterdam survey) consisted of 310 persons, who were living in 25 postal areas in the centre of Rotterdam. This study performed a non-response follow-up study (wave 4) on a sample of 177 subjects, who were approached mainly by means of house visits without prior notice.
The random sample of the second study (called the Utrecht survey) consisted of 5229 persons drawn from the municipal registry. The study in Utrecht also included a non-response follow-up study (wave 4). A random sample, stratified for age, was drawn from all non-respondents and consisted of 662 persons who were approached by telephone.
The random sample of the third study (called the large-scale Rotterdam survey) was drawn from the municipal registry in Rotterdam and consisted of 3226 persons. The large-scale Rotterdam survey did not perform a non-response follow-up study.
Measures
The outcome variable in the three studies was alcohol consumption. Alcohol consumption was measured by four questions: (1) have you drunk any alcoholic beverages in the past year?; (2) how many units of alcoholic beverages do you drink on average in a typical week?; (3) please indicate for each day in the previous week how many units of alcoholic beverages you have drunk; and (4) have you ever drunk six or more units of alcoholic beverages on 1 day in the past 6 months?
Non-respondents of the small-scale Rotterdam survey and the Utrecht survey were asked several questions about alcohol consumption. In the two non-response follow-up surveys, the interviewer asked non-respondents whether they had drunk any alcoholic beverages in the past year. If the answer was yes, these participants were asked an additional question on alcohol consumption. In the follow-up of the small-scale Rotterdam survey, this question was: have you ever drunk six or more units of alcoholic beverages on 1 day in the past 6 months? The follow-up of the Utrecht survey used the question: please indicate for each day in the previous week how many units of alcoholic beverages you consumed on each day.
We constructed several variables that provided information on the amount of alcohol consumed. The variable drinking alcohol made a distinction between abstainers and drinkers. According to the total alcohol intake in a typical week, the frequencies of drinkers were categorized as: 114 units/week, 1528 units/week and 29 units/week. This variable was called total alcohol consumption in a typical week. Based on the weekly recall question, the frequencies of the total alcohol intake consumed in the previous week were calculated (variable total alcohol consumption in previous week). The question whether the subjects had ever consumed six or more units of alcoholic beverages on one occasion and with what frequency (never; 15 times/half year; 13 times/month; 12 times/week;
3 times/week) was used for constructing the variable frequency of excessive drinking.
Analysis
Differences in the distribution of alcohol consumption between response waves were analysed by cross-tabulation. Statistical significance was estimated by chi-squared tests. A P value 0.05 was considered significant.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The survey in Utrecht had a total gross response rate of 55.5% (n = 2902). The response percentages of the three waves were 32.9, 10.7 and 11.6%, respectively. The response time of 19 persons was unknown (response percentage of 0.3%). In the non-response follow-up study of Utrecht, 370 correct telephone numbers of 662 selected non-respondents were found. The researchers contacted 254 non-respondents. During these contact attempts, 133 persons answered at least one question on alcohol. The contact rate of the follow-up was 38.4% (254/662).
The total gross response rate of the large-scale Rotterdam survey was 50.5% (n = 1630). The first wave yielded an additional 29.0%, the second wave an additional 9.5%, and the third response wave yielded an additional response of 12.1%.
Differences between early respondents, late respondents and non-respondents
Table 2 gives data on differences in alcohol consumption between early (first wave) respondents, late (third wave) respondents and non-respondents (fourth wave respondents) from the small scale Rotterdam survey. A significantly higher proportion of abstainers was found among non-respondents, than among early respondents. There were no significant differences between early respondents and non-respondents in frequencies of excessive drinking. Comparison of alcohol consumption between late respondents and non-respondents showed no significant differences, except in excessive drinking with the frequency of 15 times/half year. There was a higher proportion of excessive drinkers with a frequency of 15 times/half year among late respondents, than among non-respondents. The proportion of abstainers was also higher among non-respondents than among late respondents (third wave respondents), although the difference between these two groups was not significant.
|
|
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Some limitations in our study have to be acknowledged. One limitation is the mode of data collection. The mode of the initial surveys (i.e. the mailed questionnaire) differed from that used in the follow-up of non-respondents (i.e. face-to-face interviews/telephone interviews). Additionally, both non-response follow-ups (the small-scale Rotterdam survey and the Utrecht survey) used different modes: i.e. mainly faceto-face interviews versus telephone interviews. Because of conflicting reports in the literature, it is difficult to assess the impact of the different methods used on self-reported alcohol consumption, and on responsiveness of non-respondents in the follow-up (Rehm and Arminger, 1996; Kraus and Augustin, 2001
). There was a difference in contact rates between both our follow-up studies (83.2 vs 38.4%). Possible explanations for this difference may be the different modes used for each non-response follow-up and/or the number of contact attempts.
Besides the problem of using different modes, another limitation is the analysis itself. Most non-response follow-up studies suffer from low response rates. This is also true in our case, especially with regard to the Utrecht non-response follow-up. The response rates for questions on alcohol in the non-response follow-up study in Rotterdam and Utrecht were 79.1% (102/129) and 52.4% (133/254), respectively. This means that conclusions as to whether late respondents are more similar to non-respondents than to early respondents, rely on the assumption that the remaining non-respondents have the same pattern of alcohol consumption as fourth wave respondents. Owing to a low response rate, the results of the Utrecht non-response follow-up strongly relies on this assumption; however, the group of remaining non-respondents may become an increasingly deviant category. Although not in the field of substance use, the study by van Goor and Stuiver (1998) showed that the group of hard-core non-respondents differed more and more from the other response groups.
We were able to compare our results with three other studies in the field of alcohol and drugs. Our results showed no significant differences in alcohol consumption between first, second and third wave respondents. Similarly, Trinkoff and Storr (1997) found no significant differences in substance use rates by mailing. In the study of Ullman and Newcomb (1998)
, the pattern of results was complex and indicated few differences in behavioural low social conformity variables (i.e. frequency of use of alcohol, cigarettes and marijuana) between early and reluctant respondents and between different groups of reluctant respondents. The results of Woodruff et al.(2000)
showed significant differences in baseline smoking between on time and reluctant respondents. Smoking tobacco and marijuana use are similar to alcohol-sensitive survey topics. Ullman and Newcomb (1998)
and Woodruff et al. (2000)
included financial incentives in their reminders, which meant that they did not measure the relationship between response and substance use directly. Our results also suggest that there are no significant differences in alcohol consumption between late respondents and non-respondents, although the group differences in the Rotterdam survey were not small. Reluctant respondents and non-respondents in the study of Woodruff et al. (2000)
showed considerable similarity in their baseline smoking characteristics.
The answer as to whether the alcohol consumption distribution of late respondents can be used as a proxy for the distribution of non-respondents remains ambiguous; our results are mixed. Thus, more studies are needed to establish whether late respondents can be used as proxies for non-respondents. Also, our results show no linear patterns of differences in alcohol consumption between response waves. The question arises as to whether it may be more economical to draw a larger initial sample, rather than sending reminders in order to enhance the response rate.
To estimate non-response bias, it is advisable to do an intensive non-response follow-up on a small representative sample of non-respondents, which allows more time for each individual subject, rather than an extensive follow-up on a large sample of non-respondents.
![]() |
FOOTNOTES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Dillman, D. A. (2000) Mail and Internet Surveys: the Tailored Design Method. Wiley, New York.
Etter, J.-F. and Perneger, T. V. (1997) Analysis of non-response bias in a mailed health survey. Journal of Clinical Epidemiology 50, 11231128.[CrossRef][ISI][Medline]
Green, K. E. (1991) Reluctant respondents: differences between early, late, and nonresponders to a mail survey. Journal of Experimental Education 59, 268276.[ISI]
Hébert, R., Bravo, G., Korner-Bitensky, N. and Voyer, L. (1996) Refusal and information bias associated with postal questionnaires and face-to-face interviews in very elderly subjects. Journal of Clinical Epidemiology 49, 373381.[CrossRef][ISI][Medline]
Kellerman, S. E. and Herold, J. (2001) Physician response to surveys: a review of the literature. American Journal of Preventive Medicine 20, 6167.[Medline]
Kraus, L. and Augustin, R. (2001) Measuring alcohol consumption and alcohol-related problems: comparison of responses from self-administered questionnaires and telephone interviews. Addiction 96, 459471.[CrossRef][ISI][Medline]
Lahaut, V. M. H. C. J., Jansen, H. A. M., van de Mheen, D. and Garretsen, H. F. L. (2002) Non-response bias in a sample survey on alcohol consumption. Alcohol and Alcoholism 37, 256260.
Larroque, B., Kaminski, M., Bouvier-Colle, M.-H. and Hollebecque, V. (1999) Participation in a mail survey: role of repeated mailings and characteristics of nonrespondents among recent mothers. Paediatric and Perinatal Epidemiology 13, 218233.[CrossRef][ISI][Medline]
Lin, I. F. and Schaeffer, N. C. (1995) Using survey participants to estimate the impact of nonparticipation. Public Opinion Quarterly 59, 236258.[Abstract]
Rehm, J. and Arminger, G. (1996) Alcohol consumption in Switzerland 198793: adjusting for differential effects of assessment techniques on the analysis of trends. Addiction 91, 13351344.[CrossRef][ISI][Medline]
Schmidt, D. and Fletcher, J. (2001) Measuring response bias in survey research: an analysis of age characteristics of early respondents and resistors, pp. 113. California State University, Survey Research Center, Chico.
Trinkoff, A. M. and Storr, C. L. (1997) Collecting substance use data with an anonymous mailed survey. Drug and Alcohol Dependence 48, 18.[CrossRef][ISI][Medline]
Ullman, J. B. and Newcomb, M. D. (1998) Eager, reluctant, and nonresponders to a mailed longitudinal survey: attitudinal and substance use characteristics differentiate respondents. Journal of Applied Social Psychology 28, 357375.[ISI]
van Goor, H. and Stuiver, B. (1998) A wave-analysis of distributional bias, substantive bias and data quality in a mail survey among Dutch municipalities. Acta Politica 33, 179196.
Voogt, R. J. J., Saris, W. E. and Niemöller, B. (1998) Non-response, and the gulf between the public and politicians. Acta Politica 33, 250280.
Woodruff, S. I., Conway, T. L. and Edwards, C. C. (2000) Increasing response rates to a smoking survey for U.S. Navy enlisted women. Evaluation and the Health Professions 23, 172181.