1 Medical Research Council Tropical Epidemiology Group, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1 E7HT, UK.
2 Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK.
3 Medical Research Council Laboratories, PO Box 273, Banjul, The Gambia.
Correspondence:
Dr Shabbar Jaffar, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK. E-mail:
shabbar.jaffar{at}lshtm.ac.uk
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods Required sample sizes to achieve a study with specified power were calculated for all-cause and acute lower respiratory tract infection (ALRI) mortality for different levels of sensitivity and specificity of post-mortem questionnaires. Data from active community-based surveillance and post-mortem questionnaires collected 19891993 from the study area were used in the calculations.
Findings The mortality rate among children aged 629 months from all causes was 34.2 per 1000 child-years; 19% of deaths were attributable to ALRI. Assuming that pneumococci would be responsible for 50% of ALRI deaths and that the vaccine would cover 70% of disease serotypes and would be 90% effective against these serotypes, the expected efficacy of the vaccine would be 6.0% (19% x 50% x 70% x 90%) against all causes combined and 31.5% (50% x 70% x 90%) against deaths from ALRI. If, as suggested by various reports, the sensitivity and specificity of assigning a death to ALRI by post-mortem questionnaire are about 40% and 90% respectively, then the observed vaccine efficacy against ALRI (as classified using the post-mortem questionnaire) would fall to 20%, and the power to detect this would be reduced by approximately 40%. Furthermore, low sensitivity of diagnosis would lead to a falsely low estimate of the burden of ALRI mortality in the population and the trial might have greater power to detect a reduction in mortality from all causes combined than that estimated at the outset.
Conclusions Low sensitivity and specificity of diagnosis by post-mortem questionnaire may mean that the power of a trial to detect a reduction in all-cause mortality is similar to that to detect a reduction in ALRI mortality. Since the latter is more susceptible to bias from misclassification of cause of death, all-cause mortality may be the most suitable endpoint. Similar considerations apply to trials of interventions against other diseases for which a cause-specific endpoint is subject to substantial misclassification.
Accepted 3 December 2002
A reduction in mortality may be the most important public health measure of the efficacy of new vaccines or therapies and research trials powered to demonstrate an impact on mortality are important for informing policy. Most interventions are directed at particular causes of morbidity or mortality and thus, in general, the impact of an intervention on cause-specific mortality is more biologically meaningful than overall mortality. Also, the relative impact on cause-specific mortality will be greater and consequently trials using this endpoint are often much smaller in size than trials that use overall mortality. However, in developing countries causes of deaths are often difficult to ascertain because most deaths occur at home and funerals are held promptly. Generally, no clinical assessment is made of the cause of death. In some research studies, attempts have been made to establish the causes of death using post-mortem questionnaires (also known as verbal autopsies), in which information on signs and symptoms leading up to the death is collected from bereaved relatives by trained interviewers. The data collected are reviewed independently by a group of physicians, usually three, to assign a cause of death.1 However, the ability of this technique to diagnose accurately causes of death in African populations is questionable26 and the impact of this misclassification on the statistical power of interventions trials is unclear.
We illustrate the effects of misclassification of causes of death on the estimation of power and efficacy of a large-scale trial of a pneumococcal conjugate vaccine underway in a rural region of The Gambia.
![]() |
Background |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The study site was chosen for a number of reasons. The area is representative of a rural African setting, the population is relatively stable, coverage with vaccines in the routine Expanded Programme of Immunizations (EPI) schedule is high,10,11,13 a basic level of laboratory and clinical infrastructure is available, and large-scale research studies including the aetiology of ALRI6,9,14,15 have been conducted in the area. Between 1989 and 1993, births and deaths of children under the age of 5 years were recorded through active community surveillance and causes of death were assigned using the post-mortem questionnaire technique.10,11 The birth rate was 37.5 births per 1000 population per year. Table 1 shows the age-specific mortality rates. Deaths due to ALRI were estimated to account for 19% of the deaths.
|
Calculations of sample size were conducted using the methods described in Smith and Morrow1 with power calculated for comparing two incidence rates. A 5% two-sided significance level was used. The assumptions underlying the calculations are presented below.
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The population of the Upper River and Central River Divisions in mid-2000 was 383 000, as estimated from national census data. Approximately 14 000 births were expected to occur annually. Sample size calculations (see below) had indicated that more than 60 000 child-years of follow-up of the trial cohort might be required to achieve reasonable power to detect a significant effect of the vaccine on overall mortality. It was considered that this would be most easily achieved logistically by increasing the duration of recruitment of the trial rather than by expanding the study area.
It was estimated that children would be fully vaccinated with the pneumococcal conjugate vaccine, on average, by the age of 6 months, that approximately 850 (6.1%) infants would die annually before reaching this age, that 5% of infants would fail to receive a complete series of vaccinations, that 8% of mothers would refuse to allow their child to join the trial (figures based on the experience in a large Hib vaccine trial conducted previously in The Gambia),16 and that a further 10% of children would be lost to follow-up each year. This would leave approximately 10 600 children who would contribute fully to the post-vaccination surveillance period each year. It was planned that the recruitment for the trial would continue for a period of 3 years after the first child was fully vaccinated, and that this would be followed by a final 6-month period of follow-up. Based on the numbers of children entering the trial, the length of time until they reach the age of 30 months and their estimated mortality rates, a total of 1065 deaths, of which 203 (19%) would be attributed to an ALRI, were expected in the control group over a follow-up of 31 137 child-years.
Anticipated reduction in mortality likely to be achieved by an effective pneumococcal vaccine
The impact of a pneumococcal vaccine on overall mortality will depend on a number of factors including the proportion of deaths due to ALRI, the proportion of ALRI deaths due to pneumococci, the proportion of pneumococcal deaths attributable to serotypes included in the vaccine, and the vaccines efficacy against those serotypes. Our assumptions for the trial and the basis for these are as follows:
Assuming that 19% of deaths are attributed to ALRI, 50% of ALRI deaths are due to pneumococci, and that the vaccine covers 70% of serotypes and is 90% effective against those, then vaccination would be expected to reduce overall mortality by 6.0% (0.19 x 0.5 x 0.7 x 0.9) and ALRI attributed mortality by 31.5% (0.5 x 0.7 x 0.9). With the assumptions that 70% or 30% of ALRI deaths are attributed to pneumococci in young Gambian children (with the other assumptions remaining the same), pneumococcal vaccination might be expected to reduce overall mortality by 8.4% and 3.6% respectively and ALRI-attributed mortality by 44.1% and 18.9% respectively.
An important consideration with respect to these assumptions is that the proportion of deaths from ALRI was estimated from post-mortem questionnaires. If the sensitivity of these is low, ALRI may have accounted for a higher proportion of deaths than indicated by this estimate and, if the specificity of this technique is less than 100%, then other causes of deaths may have been incorrectly assigned as ALRI deaths. A wide combination of sensitivities and specificities are compatible with an observed ALRI death rate of 19%, each corresponding to a different true ALRI mortality rate in this population. Thus, for example, for the tabulation shown in Table 2, and for the case when the sensitivity is 0.4 and the specificity 0.9, the equations would need to satisfy the conditions below:
|
![]() |
![]() |
![]() |
![]() |
Solving these equations gives a = 129, b = 193, c = 74, and d = 669. Thus, in this scenario, the true proportion of deaths due an ALRI would have been 30% (19% as classified using the post-mortem questionnaire). If ALRI account for 30% of deaths, 50% of which are caused by pneumococci, and the vaccine covers 70% of pneumococcal serotypes and is 90% effective against these, vaccination might be expected to reduce overall mortality by 9.5% (0.3 x 0.5 x 0.7 x 0.9). If the sensitivity was 30% and the specificity 90%, then the true proportion of deaths due to an ALRI would have been 45%. Figure 1 shows the true underlying proportion of deaths due to an ALRI when 19% of deaths are recorded as such for a range of sensitivities and specificities in diagnosis.
|
|
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
If our assumptions regarding the sensitivity and specificity of the post-mortem questionnaires are correct, the overall proportion of deaths truly due to ALRI might be as high as 30%. If this is so, the impact of the vaccine on all-cause mortality would be higher than estimated at the outset and the power to detect this would be similar to that for detecting an effect against ALRI mortality as assessed through post-mortem questionnaires. The estimate of vaccine efficacy, which is crucial for calculating the burden of disease and the cost effectiveness of the vaccine, would be unbiased for all-cause mortality but biased for ALRI mortality and the extent of this bias would be unclear. A biased estimate showing, for example, a low efficacy of the vaccine against ALRI mortality may affect decisions relating to the public health priority of interventions against pneumococcal disease.
There are other advantages for using all-cause mortality as the primary endpoint. It is plausible that the benefits of the vaccine may extend beyond the target disease if, for example, it reduces measles mortality or deaths are prevented in children in whom ALRI is a secondary infection.
It is plausible that the benefits (or risks) of a vaccine may extend beyond the target disease.2225 The pneumococcal vaccine is also likely to prevent deaths from pneumococcal meningitis, which although rare, has a high case fatality rate. The verbal autopsy methodology would not be able to distinguish between different forms of meningitis and so it would be difficult to include meningitis-specific mortality in a cause-specific endpoint. Finally, the mortality from pneumococcal disease may be substantial in this setting and so a demonstration of an effect against overall mortality, even in the absence of an indirect effect against other causes of death, would have important implications for public health policy. This would greatly enhance the pressure on governments, donor agencies, and pharmaceutical companies to produce and deliver this vaccine to children in developing countries. For all these reasons, the primary endpoint chosen for the trial in The Gambia was all-cause mortality.
These findings have important implications for other trials designs involving mortality endpoints. When cause-specific mortality is used as the endpoint, it is important to consider the sensitivity and specificity of the endpoint diagnosis. Our findings also have implications for other trials where there is uncertainty about the accuracy of the measurement of the endpoint, including, for example, trials of interventions against HIV-associated opportunistic infections, or trials of vaccines against specific aetiologies of diarrhoeal disease. A recent trial of a 7-valent pneumococcal polysaccharide/protein conjugate vaccine conducted in South Africa showed an efficacy of about 20% against radiologically confirmed pneumonia with consolidation (K Klugman; personal communication). This is lower than had been expected. One interpretation of the findings is that the pneumococcus causes a lower proportion of cases of radiological pneumonia than previously considered. On the other hand, the apparently low vaccine efficacy could be biased by poor specificity of diagnosis of radiological pneumonia. The difficulty of interpreting the data from disease endpoints which are subject to misclassification argues further for a trial of this vaccine with overall mortality as a primary endpoint.
A number of potential biases need to be considered in interpreting our findings. Our assumptions that the sensitivity and specificity of the post-mortem questionnaire for diagnosing ALRI are about 40% and 90% respectively were based on a small study conducted in The Gambia,6 and a study in Kenya5 where the spectrum of disease in children and therefore the ability to predict the cause may differ from The Gambia. Further, both studies used hospital diagnoses as the gold standard against which to compare the diagnoses from the post-mortem questionnaire. It is possible that some of the hospital assessed causes of the death were incorrect and patients presenting to hospital may not have been representative of those in the wider population, which will have biased the estimates of sensitivity and specificity.
A number of other assumptions were made when calculating the power of the trial, including the proportion of eligible children entering the trial, the losses to follow-up, the likely proportion of ALRI that are caused by pneumococci, and the vaccine serotype coverage. These assumptions were based on previous experience of research conducted in the country but they may not hold during the present trial for a number of unforeseen reasons. For example, mortality may fall, perhaps from increased surveillance and improved medical care put in place for the trial, which would lower the power of the trial. We have also assumed that the efficacy of the vaccine will be 90% against pneumococcal serotypes contained in the vaccine, which we inferred from trials of a pneumococcal conjugate 7-valent vaccine against invasive disease conducted in the US.7 Whether high efficacy of the vaccine against disease would be seen in The Gambia, where the pressure of infection from pneumococci, the distribution of other pathogens, the mechanisms for vaccine delivery, and host immune responses may differ from the US population, is not clear. It is also plausible that the efficacy of the pneumococcal vaccine against mortality is lower than for disease if, for example, the vaccine does not protect against non-invasive ALRI disease, as is the case for the polysaccharide pneumococcal vaccine.26 In this event, the sample size of our trial may prove to be inadequate. However, for the purpose of illustrating the effect of misclassification, we assumed a vaccine efficacy of 90% in this paper.
Our findings highlight the difficulties of calculating sample sizes even in a setting such as ours where surveillance studies for disease and mortality were conducted over several years in preparation for this trial. One important strategy for dealing with uncertainty in the degree of misclassification of the endpoint is to conduct an interim analysis and re-estimate sample sizes at that time. This should be done using a low significance so that it does not affect appreciably the significance in the final analysis.27
In summary, low sensitivity and specificity of diagnosis by post-mortem questionnaire may mean that the power of a trial to detect a reduction in all-cause mortality is similar to that to detect a reduction in cause-specific mortality. Since the latter is more susceptible to bias from misclassification of causes of death, all-cause mortality may be the most suitable endpoint. Our investigations highlight the importance of taking diagnostic sensitivity and specificity into account when designing efficacy trials against disease-specific endpoints.
![]() |
Appendum |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
Acknowledgments |
---|
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
2 Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ. Verbal autopsies for adult deaths: their development and validation in a multicentre study. Trop Med Int Health 1998;3:43646.[ISI][Medline]
3 Quigley MA, Chandramohan D, Rodrigues LC. Diagnostic accuracy of physician review, expert algorithms and data-derived algorithms in adult verbal autopsies. Int J Epidemiol 1999;28:108187.[Abstract]
4 Todd JE, De Francisco A, ODempsey TJ, Greenwood BM. The limitations of verbal autopsy in a malaria-endemic region. Ann Trop Paediatr 1994;14:3136.[ISI][Medline]
5 Snow RW, Armstrong JR, Forster D et al. Childhood deaths in Africa: uses and limitations of verbal autopsies. Lancet 1992;340:35155.[CrossRef][ISI][Medline]
6 DAlessandro U. An epidemiological evaluation of the impact of a national impregnated bednet programme on mortality and malaria morbidity in children and on the outcome of pregnancy in primigravidae in the Gambia, West Africa. In Department of Infectious and Tropical Diseases. 1996, London School of Hygiene and Tropical Medicine: London.
7 Black S, Shinefield H, Fireman B et al. Efficacy, safety and immunogenicity of heptavalent pneumococcal conjugate vaccine in children. Northern California Kaiser Permanente Vaccine Study Center Group. Pediatr Infect Dis J 2000;19:18795.[CrossRef][ISI][Medline]
8 Shinefield H, Black S. Efficacy of pneumococcal conjugate vaccines in large scale field trials. Pediatr Infect Dis J 2000;19:39497.[CrossRef][ISI][Medline]
9 Obaro S, Adegbola SK, Chang RA et al. Safety and immunogenicity of a nonavalent pneumococcal vaccine conjugated to CRM197 administered simultaneously but in a separate syringe with diphtheria, tetanus and pertussis vaccines in Gambian infants. Pediatr Infect Dis J 2000;19:46369.[CrossRef][ISI][Medline]
10 De Francisco A, Hall AJ, Schellenberg JR, Greenwood AM, Greenwood BM. The pattern of infant and childhood mortality in Upper River Division, The Gambia. Ann Trop Paediatr 1993;13:34552.[ISI][Medline]
11 Jaffar S, Leach S, Greenwood A et al. Changes in the pattern of infant and childhood mortality in upper river division, The Gambia, from 1989 to 1993. Trop Med Int Health 1997;2:2837.[CrossRef][ISI][Medline]
12 Jaffar S, Leach A, Hall AJ et al. Preparation for a pneumococcal vaccine trial in The Gambia: individual or community randomisation? Vaccine 1999;18:63340.[CrossRef][ISI][Medline]
13 De Francisco A, Schellenberg JA, Hall AJ et al. Comparison of mortality between villages with and without Primary Health Care workers in Upper River Division, The Gambia. J Trop Med Hyg 1994;97:6974.[ISI][Medline]
14 ODempsey TJD, McArdle TJ, Lloyd-Evans TF et al. Pneumococcal disease among children in a rural area of West Africa. Pediatr Infect Dis J 1996;15:43137.[CrossRef][ISI][Medline]
15 DAlessandro U, Leach A, Drakeley CJ et al. Efficacy trial of malaria vaccine SPf66 in Gambian infants. Lancet 1995;346:46267.[ISI][Medline]
16 Mulholland K, Hilton S, Adegbola R et al. Randomised trial of Haemophilus influenzae type-b tetanus protein conjugate vaccine for prevention of pneumonia and meningitis in Gambian infants. Lancet 1997;349:119197.[CrossRef][ISI][Medline]
17 Usen S, Adegbola R, Mulholland K et al. Epidemiology of invasive pneumococcal disease in the Western Region, The Gambia. Pediatr Infect Dis J 1998;17:2328.[CrossRef][ISI][Medline]
18 ODempsey TJ, McArdle TF, Laurence BE et al. Overlap in the clinical features of pneumonia and malaria in African children. Trans R Soc Trop Med Hyg 1993;87:66265.[ISI][Medline]
19 Anonymous. Measurement of overall and cause-specific mortality in infants and children: memorandum from a WHO/UNICEF meeting. Bull World Health Org 1994;72:70713.[ISI][Medline]
20 Benara SK, Singh P. Validity of causes of infant death by verbal autopsy. Indian J Pediatr 1999;66:64750.[Medline]
21 Rodriguez L, Reyes H, Tome P et al. Validation of the verbal autopsy method to ascertain acute respiratory infection as cause of death. Indian J Pediatr 1998;65:57984.[Medline]
22 Hall AJ, Aaby P. Tropical trials and tribulations. Int J Epidemiol 1990; 19:77781.[ISI][Medline]
23 Aaby P, Samb B, Simondon F et al. Non-specific beneficial effect of measles immunisation: analysis of mortality studies from developing countries. BMJ 1995;311:48185.
24 Aaby P, Shaheen SO, Heyes CB et al. Early BCG vaccination and reduction in atopy in Guinea-Bissau. Clin Exp Allergy 2000;30:64450.[CrossRef][ISI][Medline]
25 Kristensen I, Aaby P, Jensen H. Routine vaccinations and child survival: follow up study in Guinea-Bissau, West Africa. BMJ 2000; 321:143538.
26 Watson L, Wilson BJ, Waugh N. Pneumococcal polysaccharide vaccine: a systematic review of clinical effectiveness in adults. Vaccine 2002;20:216673.[CrossRef][ISI][Medline]
27 Peto R, Pike MC, Armitage P et al. Design and analysis of randomized clinical trials requiring prolonged observation of each patient. I. Introduction and design. Br J Cancer 1976;34:585612.[ISI][Medline]