Forward projection—using critical appraisal in the design of studies

Mark Elwood

National Cancer Control Initiative, 1 Rathdowne St, Carlton, Victoria 3053, Australia. E-mail: melwood{at}ncci.org.au

Traditional courses of epidemiology usually start with descriptive epidemiology, moving to types of study design, and often ending with the presentation of results and some statistics. More recent epidemiological graduates, particularly those with a clinical background, may have covered the subject the opposite way, by starting with critical appraisal of studies in the context of evidence-based medicine, emphasizing the interpretation of study results. Either way, the process of conceptualizing and designing studies is often separated from that of interpreting other people’s studies in published literature. I have found it helpful in discussing study design with students or with health professionals to use the principles of critical appraisal in study design.

It is useful early in the work, when a preliminary approach has been outlined. Let us take an epidemiological issue such as whether exposure to electric and magnetic fields from power lines increases the risk of depressive illness. From that point, an outline of the study design may be developed, for example a case-control study based on patients in a large general practice looking at place of residence in proximity to power lines. Some approximate idea of the size of the study and time framework may be assumed, along with some preliminary ideas of accessing the patient information from medical records, approaching the subjects, and carrying out an interview about place of residence. A control group could be drawn from other patients in the same practices. At this point I have found it helpful to encourage the study designer to imagine themselves giving a seminar on the results of the study in 1 or 2 years time to a friendly but critical audience of his or her professional colleagues and interested laypeople. And also, to imagine themselves in the audience of such a seminar at which they are presented with results which, let us say, show a strong association with increased rates of depression in those living close to power lines.

If they have taken a course in critical appraisal, they of course will know what to do next. They will use one of the many schemes for critical appraisal, such as that shown in Table 1Go.1 This one starts by describing the environment and study design used and the main result. Then it considers the possible non-causal explanations for that result: observation bias, confounding and chance. This will force the investigator to think about the quality of their data and whether there is likely to be observation bias. Are the subjects with depression and the control subjects likely to respond to the questionnaire in the same way? It will make them think of ways to make the study design strong enough to provide results which are not compromised by observation bias, but also how this could be demonstrated. Would the questionnaire be helped by including some factors which we expect not to be different in the two groups, as a way of testing this? Confounding is also likely to be given new light by this procedure. What factors might be associated with the subjects with depression in this study? What factors could be related to residence near power lines? Factors needing further consideration will be those appearing on both lists, or appearing as very strong factors on one of the lists. And how about chance variation; will the power of the study be sufficient that a result showing an association which would be clinically important, perhaps a relative risk of 1.5 or 2, will be statistically significant? Will analyses be needed for certain subgroups, and what will that do to the power? The overall question is that if the result shows an association, can that be interpreted as a causal relationship?


View this table:
[in this window]
[in a new window]
 
Table 1 Scheme for critical assessment of studies to assess causality, from Elwood, 1998, applied to study design
 
It is worth at this point stopping and then considering the same issues where the result of the study shows no difference. And again, if the results show an effect in the opposite direction. This particular exercise is very valuable with an investigator who believes or hopes that the results of the study will support their preconceptions. If you were listening to this seminar, and the results were opposite to what you expect, what questions would you have? Will the study be robust enough in its methods for you to change your mind? If not, why not? This will perhaps throw up some major weaknesses in the study and make it clear that if there are such weaknesses, why would we expect a result which fits our previous ideas to be accepted by others? I was once asked by a cancer charity to help them design a study to show that their education programme was effective. After some discussion, I asked ‘What will you do if the programme doesn’t appear effective?’ This caused consternation, and eventually they decided not to do the study as they could not accept this possibility. In similar discussions of a study to assess a clinician’s pet treatment option, the initial response is often ‘I’ll keep using my treatment’, as this is only one study, or a small study, or something else, which obviously raises questions about the value of the assessment.

The positive features of causation, based on the Bradford Hill criteria, are of course not absolute but may often be helpful. What strength of relationship would be needed to be convincing? This takes us back to the power of the study. Do we expect a dose-relationship, and if so is the study designed in such a way that it can be demonstrated; for example, with length of residence? Are there aspects in which consistency or specificity would help, and could these be brought into the study design; for example, is the result consistent in men and women? Power lines may be ugly intrusive structures; would it be worth asking about proximity to other such structures such as television masts, high buildings or industrial facilities?

The section on generalizability concerns whether the study population will allow us to apply the results to more general populations. If the study is done in the district and practices under consideration, are the results going to be applicable elsewhere in the country or beyond it? If the subjects are in a certain age or socioeconomic range are the results going to be applicable further? This may cause a re-think of the study setting.

There will usually be other evidence already available on the topic, and this will force us to ask if the results will be valuable in comparison with the available literature. This gets at the heart of the issue of whether the study is worth doing. I well remember a clinical meeting in which a large immaculately performed clinical trial, involving 2000 patients and published in The New England Journal of Medicine, was discussed, and one comment given in perfect seriousness was that these results were interesting, but we should do a local study to confirm them; this in a hospital with some 50 relevant patients per year. This issue may open up questions of whether a wider and more detailed literature review is needed, whether collaborations should be sought with other groups, and whether more inquiries should be made about studies which were on-going elsewhere.

The ‘seminar’ exercise can be done mentally, or better, in conversation with colleagues, in various degrees of detail, and at various points in the design process. It will encourage thinking about the format of the results, considering what the results might look like in the final study, what further analyses will be needed, and what points will need to be considered in the discussion and therefore addressed in the study design. After such consideration the methods can be written down for the protocol and also for the future presentation or publication. Some teachers suggest that the paper can be written in draft form, with imaginary results and useful discussion, although I have found that the more informal process of imagining a future seminar is more helpful and more enjoyable. This approach works very well in small group work with students or professional colleagues. Obvious role-play situations (the cynic, the evangelist, the funder, the journalist, the bureaucrat, ...) come to mind. Involving those with expertise in the subject matter may be extremely valuable; in the example given, a general practitioner, a psychiatrist, and an electrical engineer would all have new insights.

Reference

1 Elwood JM. Critical Appraisal of Epidemiological Studies and Clinical Trials. Oxford: Oxford University Press, 1998.





This Article
Extract
FREE Full Text (PDF)
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Request Permissions
Google Scholar
Articles by Elwood, M.
PubMed
PubMed Citation
Articles by Elwood, M.