1 Division of Preventive Oncology, Cancer Care Ontario, Toronto, Ontario, Canada
2 Department of Public Health Sciences, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
3 Samuel Lunenfeld Research Institute, Mount Sinai Hospital, Toronto, Ontario, Canada
4 PICEPS Consultants, Inc., Ajax, Ontario, Canada
5 Department of Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
6 Department of Oncology, Faculty of Medicine, McGill University, Montreal, Quebec, Canada
7 Current affiliation: Canadian Institute for Health Information, Toronto, Ontario, Canada
Correspondence to Dr. Loraine D. Marrett, Division of Preventive Oncology, Cancer Care Ontario, 620 University Avenue, Toronto, Ontario M5G 2L7, Canada (e-mail: loraine.marrett{at}cancercare.on.ca).
Received for publication July 20, 2004. Accepted for publication January 6, 2005.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
bias (epidemiology); case-control studies; melanoma; mental recall; questionnaires; research design
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Confidence is the extent to which a person believes the information s/he has provided is correct (7), while accuracy is a measure of how close the information provided is to the truth (8
). Several eyewitness crime studies have shown that self-reported confidence ratings are correlated with recall accuracy (9
13
), so that the more confident a person is in the accuracy of a response, the higher the probability that the response is accurate. If the confidence-accuracy relation is true for recalling a past exposure, then confidence ratings may be useful in assessing the quality of recalled data.
Some studies suggest that conversing with other people about past events or self-reflecting on past events increases confidence in reporting (1419
). Since people diagnosed with melanoma may be more likely to have engaged in these behaviors, their recall of past events may have been affected, resulting in greater confidence in reporting in comparison with people without melanoma. Thus, if confidence in reporting previous exposures to the sun differs between cases and controls, there may be recall bias.
In the current study, we evaluated the association between case-control status and self-reported confidence as an indirect means of examining recall bias in reported time spent outdoors in various activities during early life summers. We also explored other predictors of self-reported confidence to better understand the role of confidence ratings in studies requiring recall of past behaviors.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Case and control ascertainment
People diagnosed with cutaneous malignant melanoma were identified through pathology reports provided to the Ontario Cancer Registry, a comprehensive population-based cancer registry covering the province of Ontario, Canada (20). Eligible cases were defined as Caucasian Ontario residents who were aged 2044 years when they were first diagnosed with melanoma (including melanoma in situ) between September 2000 and June 2002 and who were able to complete a telephone interview in English. Physician consent was obtained before study staff contacted the potential case participant.
Population-based controls were selected from the Ontario general population using random digit dialing and were quota sampled to have the same age and sex distribution as that expected for cases. Recruiters screened potential controls for eligibility and found that the majority of those contacted were ineligible.
Data collection
Potential participants were mailed an information package containing 1) a description of the study; 2) a phenotypic and demographic questionnaire; 3) an early life events calendar ("biography"); 4) three sample biographies, including the events calendar as completed by fictitious respondents; 5) a list of outside activities to assist recall; and 6) a stamped, self-addressed envelope. Subjects completed and mailed back the questionnaire and the early life events calendar (completed with reference to the provided samples and activity list) in the envelope provided.
Once the two self-administered documents were received at the study office, telephone interviews were conducted from the Institute of Social Research at York University (86 percent) or from Cancer Care Ontario (14 percent), using a semistructured questionnaire to collect information about sun exposure.
This substudy focused on the reported amount of time spent outdoors during the summer months from age 6 years to age 18 years, which is when Ontario children generally receive the majority of their sun exposure. Ontario children generally do not have school in the summer, and Ontario usually receives more sunlight during the summer months than throughout the rest of the year. Interviewers used the completed early life events calendar to help respondents recall their outdoor summer activities (e.g., chores, vacations, and "hanging out"). This dialogue between the interviewer and the respondent encouraged a conversational approach to data-taking. For each summer activity, respondents identified periods of stable outdoor exposure. That is, within each activity, respondents defined periods of life that had relatively constant patterns of outdoor exposure. For each such stable outdoor activity period identified, respondents were asked to estimate the amount of time they had spent outdoors between the hours of 9 a.m. and 5 p.m. They were then asked, "How confident are you that the amount of time you said you spent outdoors [activity] is accurate? Can you rate it on a scale of 1 to 5, where 1 is not at all confident and 5 is very confident?".
A total of 368 eligible cases were identified, and physician consent was obtained for 311 (85 percent). Of the 311 eligible cases who were sent packages, 196 (53 percent of identified cases) returned completed packages to the study office, 33 refused to participate, and 82 were nonresponders. Among the persons who returned packages, 182 (49 percent of identified cases) completed telephone interviews, seven were nonresponders, and seven refused to participate. Of these interviews, questions about confidence ratings were included in only 141 interviews conducted between October 2002 and April 2003.
Most households contacted during control recruitment contained no one who met the eligibility criteria; approximately one call in 20 yielded an eligible person who agreed to be mailed an information package. Unfortunately, no information was collected on the other 19 households (i.e., whether there was no one eligible or an eligible person declined to receive a package). Thus, the control response rates reported are overestimated to an unknown degree. A total of 309 eligible controls agreed to receive information packages, of whom 185 (60 percent of controls who agreed to receive packages) returned completed packages to the study office, 41 refused to participate, and 83 were nonresponders. Among the persons who returned packages, 175 (57 percent of controls who agreed to receive packages) competed telephone interviews, four refused to participate, and six were nonresponders. Of these interviews, questions about confidence ratings were included in only 144 interviews conducted during data collection for this substudy. We excluded data obtained from one control who declined to provide complete demographic information, which resulted in 143 interviews for analysis.
Stratification
Data were analyzed in two age strata: activities reported for ages 611 years and activities reported for ages 1218 years. For activities that overlapped the two age strata, the appropriate amount of time was allocated to each stratum, so that the same respondents were represented in both strata. Data were stratified because more recent events are likely to be recalled more confidently than events occurring farther back in time (21, 22
). A cutpoint of age 12 years was chosen because the number, type, and variety of activities started to increase at this age.
Variables for analysis
Research suggests that cognitive effort and confidence are related (10, 17
, 23
, 24
). The total number of stable outdoor activity periods a respondent reported in each age stratum was used as an indirect measure of cognitive effort. This was modeled as a binary variable, using the median as the cutpoint.
Questions about types of activities performed were asked in an open-ended format. In order to include activity as a variable for analysis, we categorized activities into five groups. For example, "playing" and "hanging out" both describe the same general activity and were grouped together.
The study outcome was self-reported confidence ratings for the amount of time spent outdoors during activities performed in early life summers. Within each stratum and activity group, a respondent's confidence ratings were averaged, with weightings determined by the number of summers each activity in the group was performed. That is, one mean confidence score was created per person-activity, and the number of such scores for a subject varied depending on the number of activity groups in which s/he engaged (out of a possible five activity groups). Because mean confidence scores had a nonnormal distribution, they were dichotomized into "more confident" (4) and "less confident" (<4). The dichotomized weighted mean confidence score was modeled as a binary variable.
Time spent outdoors may confound the relation between confidence and case-control status, because sun exposure is an important risk factor for melanoma (2527
). It is also possible that the more time spent outdoors in an activity, the more confident a person becomes in estimating the activity duration. The variable "cumulative time spent outdoors" was calculated by summing the numbers of hours a person had spent outdoors in each activity group for each age stratum and dividing these totals into quartiles.
Age may be a predictor of confidence. To explore this, we defined age as age at diagnosis for cases and age at time of interview for controls. Sex was of interest because little autobiographic research has been conducted on sex differences with respect to confidence (28), and what little there is has identified no clear and consistent differences (29
31
). The highest level of education subjects had attained was also included as a covariate, because few or no studies have examined the association between confidence in recall of past exposures and education.
Statistical analysis
To account for the multiple confidence scores for each subject, we used the generalized estimating equations extension of logistic regression to model the binary confidence outcome (32). An exchangeable correlation matrix was chosen, which assumes that the correlation between any two confidence ratings is constant. Robust estimators of variance were used to construct confidence intervals for odds ratios, and robust score tests were conducted to assess the contribution of each independent variable to the model. All generalized estimating equations analyses were carried out using the PROC GENMOD procedure in SAS, version 8.2 (33
).
Distribution of confidence ratings
A total of 284 respondents provided 2,651 reports of stable outdoor activity periods (figure 1). There were 1,166 reported stable outdoor activity periods in the age stratum 611 years; 827 of these periods were solely within this age range, while 339 were in both age strata. One control reported activities only for ages 1218 years, which resulted in our having 283 respondents in the 611 age stratum. The median number of stable outdoor activity periods reported per person was three (range, 113). Five reports in which participants responded "don't know" to the question on amount of time spent outdoors and six reports with missing confidence ratings were excluded. The remaining 1,155 reports of stable outdoor activity periods and their corresponding confidence ratings were grouped as described above, resulting in 574 person-activities for the age stratum 611 years.
|
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
|
|
|
|
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Since there was little difference between cases and controls with respect to confidence, there may have been little recall bias in reported time spent outdoors in this study. The RESE methods of using an early life events calendar, having a conversational approach to interviewing, and helping respondents identify similar episodes of outdoor time may have encouraged more complete and confident recall for cases and controls equally.
Previous studies of sun exposure have used various methods to assess recall bias. For example, Berwick and Chen (34) asked respondents about their history of sunburns, reinterviewed respondents 13 years later, and then compared the two reports. English et al. (35
) also conducted a test-retest reproducibility study on reported measurements of sun exposure. Weinstock et al. (2
) compared female nurses' self-assessment of their ability to tan before and after being diagnosed with melanoma. However, this question was worded differently in the two questionnaires. Based on the theoretical framework from psychological studies, the current study presents an alternative method of assessing recall bias: self-reported confidence ratings. Unlike the above studies, in which people were interviewed on more than one occasion, confidence ratings can save investigators time and expense by being incorporated into the initial interview.
All activity groups were associated with greater confidence in reporting duration of outdoor activity than was "hanging out." "Hanging out" activities were described by RESE respondents as being many, vague, and unstructured (e.g., engaging in pick-up sports and playing) (Relova et al., Cancer Care Ontario, unpublished manuscript). In comparison, activities such as "camp," "trips/vacations," and "work/chores" were more specific and structured. Psychological studies show that memories of specific and regularly repeated (structured) daily activities (e.g., doing laundry or eating breakfast) are stored in a permanent part of the brain's memory (36), and people find it easier to recall and report how often they engaged in these activities (37
). Because the demand for cognitive effort is negatively correlated with confidence in reporting (10
, 17
, 23
, 24
), it is not surprising that our findings showed that the durations of specific and structured activities were reported more confidently than the durations of vague and unstructured activities.
We also found that the greater the amount of time spent outside in an activity, the greater the confidence in recall. In the cognitive psychology literature, spending more time on a task has been shown to increase confidence in reporting its duration (38). In addition, structured activities were often repeated, and this repetition increased the cumulative amount of time spent outdoors. Thus, the inherent structure of some activities is an additional explanation for the association between cumulative time spent outdoors and confidence.
An additional finding was that the greater the number of stable outdoor activity periods, the lower the confidence in reporting. It is likely that difficulty in recalling and quantifying specific activity-period durations increased with increasing number of activity periods. For example, a respondent who had engaged in many activities had to focus on and provide time estimates for each activity period, which demanded more cognitive effort in comparison with someone who had engaged in fewer activities. As we noted above, as demand for cognitive effort increases, confidence diminishes.
One unique feature of the RESE instrument is the respondent's ability to identify as many (or as few) stable outdoor activity periods as needed. Although those respondents reporting a greater number of stable outdoor activity periods were less confident, this need not imply that they, or this method, provided less accurate data. An alternative interpretation is that these people led busier lives, thus requiring more recall effort regardless of how outdoor time data were collected. More research is needed to explore the associations among the effects of asking respondents to estimate time spent outdoors multiple times, self-reported confidence, and accuracy.
There was little difference in confidence for the younger age groups (i.e., ages 2029 years and 3039 years) when they were compared with the older age group (i.e., persons aged 4044 years). This differs from other studies which have shown that the longer the retention interval (age at recall minus age at event), the less confident people become (21, 22
). In the current study, the retention interval increased with the subject's age, so one might have expected that older subjects would be less confident than younger subjects. One possible explanation for our finding is that older people may have reminisced about the past more often than younger people and that this increased mental rehearsal may have increased older subjects' confidence in reporting (39
, 40
).
Eyewitness studies have found a positive correlation between confidence and accuracy (913
). Our findings on confidence are consistent with those of eyewitness studies (cognitive effort and retention interval) and psychological studies (structured and nonstructured activities and cumulative time spent outdoors). Therefore, in the context of recalling a past exposure, it may be plausible to infer that confidence is a predictor of recall accuracy. In fact, eyewitness studies involving recall tasks (e.g., recalling the perpetrator's clothing, crime location, and so forth) tend to have higher confidence-accuracy correlations than those involving recognition tasks (e.g., identifying perpetrators in a lineup or selecting a response when alternative responses are provided) (10
, 24
).
Recall of life events is more dynamic than recall of laboratory eyewitness events. First, in comparison with eyewitness studies, real-life events take place outside of a laboratory, and people are active participants in their lives. These undoubtedly distinct experiences stimulate different senses, emotions, and perspectives. Thus, the formation of memories, the characteristics of memories recalled, and perhaps confidence in memory recall are different. In fact, the vividness and quality of memories recalled have been shown to influence confidence in reporting (21). Second, recall of personal events can literally span a lifetime, whereas eyewitness studies generally take place during a period of 1 or 2 hours. The longer the retention interval, the less confident people become (21
, 22
) and the higher the chances that social influences and postevent information will alter a person's initial assessment of confidence (13
). Although recall of life events is more complex than recall of laboratory eyewitness events, eyewitness studies still provide valuable insight into how people recall past events and the factors influencing self-reported confidence.
One of this study's limitations was the low response rate for both cases and controls. Because we did not collect information on controls who were unwilling to receive study packages, we were unable to compare the characteristics of control responders and nonresponders. We also did not know whether responders and nonresponders were likely to differ with respect to how confident they were in estimating amounts of time spent outdoors. To our knowledge, this is the first study to investigate self-reported confidence in the context of early life sun exposure, and more studies are needed to investigate the effects, if any, of low response rates on study findings.
Respondents may have reported that they were very confident about their responses to make it appear that they remembered the past well. Such social desirability bias seems unlikely to have differed between cases and controls, and therefore would not invalidate the study findings.
Finally, in this study, we were not trying to elucidate the etiology of melanoma or to evaluate sun exposure measurements. Instead, the purpose of this study was to use confidence ratings to understand and qualify the strengths and limitations of the RESE instrument. Other researchers are encouraged to apply confidence ratings to question(s) related to specific measures of interest.
In conclusion, the results of this study suggest that there is no association between case-control status and confidence in the reporting of time spent outdoors during activities conducted in early life summers. For both age strata (ages 611 years and 1218 years), activity group was a predictor of self-reported confidence, and as the number of stable outdoor activity periods reported increased, confidence decreased. Cumulative time spent outdoors was a predictor only for the age stratum 1218 years.
Since case-control status was not a predictor of self-reported confidence, recall bias may have been minimized. Perhaps the design of the RESE Study, which included an early life events calendar, a conversational approach to interviewing, and identification of stable outdoor activity periods, encouraged both cases and controls to reflect on their past sun exposure, thereby minimizing bias. We suggest that epidemiologists consider using self-reported confidence ratings to assess the quality of data collected and its impact on the validity of study results.
![]() |
ACKNOWLEDGMENTS |
---|
This substudy was partly funded by the National Cancer Institute of Canada (Programme of Research in Environmental Etiology of Cancer). The main RESE Study was funded by the US National Cancer Institute (grant 1 R03 CA88206-01).
Conflict of interest: none declared.
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|