Accuracy of cause-of-death coding in Taiwan: types of miscoding and effects on mortality statistics

Tsung-Hsueh Lua, Meng-Chih Leeb and Ming-Chih Choub

a Department of Public Health, Chung Shan Medical and Dental College, Taichung, Taiwan.
b Department of Family Medicine, Chung Shan Medical and Dental College Hospital, Taichung, Taiwan.

Reprint requests to: Dr Ming-Chih Chou, Department of Family Medicine, Chung Shan Medical and Dental College Hospital, 23, Section 1, Taichung Kang Road, Taichung 403, Taiwan. E-mail: robertlu{at}ms1.hinet.net


    Abstract
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
Background The objectives of this study were to assess the accuracy of cause-of-death coding, determine the extent to which coders follow the selection rules of coding set out in the International Classification of Diseases, 9th Revision (ICD-9), and the effects of miscoding on mortality statistics in Taiwan.

Method A systematic sample of 5621 death certificates was reviewed. The underlying cause of death (UCD) selected by the reviewer for each death certificate was compared with that selected by the original coder. The UCD was selected according to ACME (Automated Classification of Medical Entities) Decision TablesGo.


View this table:
[in this window]
[in a new window]
 
 
Results The overall agreement rates between the reviewer and coders according to the three-digit and two-digit categories of ICD-9 were 80.9% and 83.9%, respectively. Good agreement was found for malignant neoplasms (kappa = 0.94) and injuries and poisoning (kappa = 0.97), but there was poor agreement for nephrotic diseases (kappa = 0.74), hypertension-related diseases (kappa = 0.74), and cerebral infarction (kappa = 0.77). Reasons for disagreements included disagreement in nomenclature (42.8%), inappropriate judgement of causal relationships (41.5%), and incorrect interpretation of Selection Rule 3 and Modification Rules (15.7%).

Conclusion This study showed various levels of agreement for different diseases between the reviewer and the original coders in selection of the UCD. Owing to the ‘compensatory effect of errors’, the national mortality statistics were not affected significantly. The national administration should undertake routine internal studies to control the quality of UCD coding practices.

Keywords Cause of death, coding, ICD-9, death certificates, mortality, Taiwan

Accepted 15 July 1999


    Introduction
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
Cause-of-death statistics are widely used to monitor the health of general populations and are consequently important for health planning and setting priorities for disease prevention.1 Accurate ascertainment of cause of death is also very important in clinical trials and outcome review studies.2 The International Classification of Diseases (ICD)3 was developed by the World Health Organization to standardize cause-of-death coding practices and thereby allow valid comparisons of cause-specific mortality among various countries.

Despite the detailed and specific rules provided in the ICD for coding death certificate data, coding practices still vary substantially among countries.4–11 Some studies have focused on the coding problems related to specific diseases in particular counties, such as hypertension,12 rheumatoid arthritis,13,14 myocardial infarction,15 Down syndrome16 and short gestation.17 Few studies have discussed the types of coding errors and the reasons they occur,4,6,14 and only one study has provided corrected mortality rates after adjusting for coding discrepancies.6

The objectives of this study were to assess the accuracy of cause-of-death coding in Taiwan and to determine the extent to which coders follow the selection rules of coding set out in the ICD. We also looked at which diseases were overcoded and which were undercoded, and the effects of miscoding on mortality statistics in Taiwan.


    Methods
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
All the death certificates issued by doctors in Taiwan are sent to the Taiwan Provincial Department Health, Office of Statistics, for tabulation. Five coders in the Office of Statistics are responsible for selecting the underlying cause of death (UCD) for each death certificate.

In 1994, 112 238 death certificates were issued in Taiwan. We chose to use a 5% sample for this study because we determined that this would yield sufficient numbers (i.e. >5) in each cross-tabulation cell, especially in the analysis by ICD chapter. Death certificates were systematically sampled (1 of every 20). Because there was little possibility of cyclic characteristics in this series of numbers, we did not use a randomly selected starting point. Thus, the first author reviewed 5621 death certificates and assigned a UCD for each, with strict adherence to the International Classification of Diseases, 9th Revision (ICD-9) selection rules (Appendix). Because the interpretation of Selection Rule 3 and Rule 7 is highly controversial, ACME (Automated Classification of Medical Entities) Decision TablesGo18 were used to determine whether they were applied properly. The ACME Decision TablesGo were also used to determine whether the causal sequence of causes of death was ‘highly improbable’ or not.

Before the formal review, 200 death certificates randomly selected from the 5621 certificates were reviewed by the first author on two occasions, one month apart. The UCD were not concordant between the two reviews for only three of the 200 death certificates. The reasons for the discrepancies were inconsistent interpretation of Rule 3 for septic shock and sepsis. The interpretations of these situations were unified for the formal review.

The UCD selected by the reviewer were then compared with the UCD selected by the original coders. The UCD selected by the reviewer was viewed as the ‘gold standard’ to calculate the agreement rate, false-positive rate, false-negative rate, and kappa values.19 The methods of calculating different indicators of agreement are illustrated as follows:

Chi-square analysis was used to test the differences in agreement rates according to various characteristics (sex and age of the deceased, number of diagnoses per death certificate, different coders). We further classified the disagreements into different types according to the reason for disagreements. The death rates estimated according to the reviewer and the original coders were compared to evaluate the effects of coding errors on mortality statistics. Age-adjusted mortality rate ratios (reviewer/original coder) and 95% CI20 were calculated to determine the extent of overcoding (mortality rate ratio <1) and undercoding (mortality rate ratio >1) for main causes of death.


    Results
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
The sample size (5621) represents 5% of the total deaths in Taiwan in 1994. No statistically significant difference was found between the study sample and the national data, when compared by percentage of deaths according to age-sex intervals and main causes of death categories (not shown).

Factors associated with the level of agreement
Overall, the agreement rates between original coders and reviewer according to ICD-9 three-digit and two-digit categories were 80.9% and 83.9%, respectively. The agreement rates decreased as the age of the deceased increased, and as the number of diagnoses entered on the death certificate increased (Table 1Go). According to the two-digit categories, the agreement rates varied from 82.4% to 86.6% between the reviewer and each of the five coders. The variation increased (78.2–84.2%) if the three-digit category was used.


View this table:
[in this window]
[in a new window]
 
Table 1 Agreement in underlying cause of death (UCD) selection between reviewer and original coder by different characteristics, Taiwan, 1994a
 
Indicators of levels of agreement
A cross tabulation of UCD selection between the reviewer and the original coders by ICD-9 chapters is illustrated in Table 2Go. In 23 cases in which the reviewer determined the UCD to be a circulatory disease (most of which were cerebrovascular disorders, CVD), the original coder determined an endocrine disease (diabetes in most cases) to be the UCD. In contrast, for 46 death certificates in which the reviewer chose an endocrine disease (diabetes in most cases) as the UCD, the original coder selected a circulatory disease (most of which were CVD) as the UCD. In addition, the reviewer and coders often disagreed on the selection of liver cirrhosis versus liver cancer (21 cases) and chronic obstructive pulmonary disease (COPD) versus CVD (44 cases) as the UCD. Disagreement in nomenclature was another source of disagreement. For example, the reviewer coded ‘septic shock’ as 785.5 (shock without mention of trauma), and ‘cardiac arrest’ or ‘heart asystole’ as 799.9 (other unknown and unspecified causes of morbidity and mortality), while some of the original coders coded them as 038 (septicaemia) and 427 (cardiac dysrhythmia), respectively.


View this table:
[in this window]
[in a new window]
 
Table 2 Cross tabulation of underlying cause of death (UCD) selected by the reviewer and original coders according to ICD-9a Chapters, in Taiwan, 1994b
 
There were six cases in which the UCD selected by the reviewer was pregnancy-related death (ICD-9 Chapter XI), while there were only three cases of pregnancy-related death according to the original coders' selections. Thus, the maternal mortality rate estimated according to the reviewer was twice that estimated according to the original coders (Table 2Go).

The agreement was good for malignant neoplasms (kappa = 0.94) and injuries and poisoning (kappa = 0.97), and poor for nephrotic diseases (kappa = 0.74), hypertension-related diseases (kappa = 0.74), and cerebral infarction (kappa = 0.77) by main causes of death (Table 3Go). The false-positive rate represents the degree of overcoding of a disease by the original coder as judged the reviewer. The false-negative rate, on the other hand, expresses the degree of undercoding of a disease by the original coder in comparison with the reviewer. Liver cancer (20%), cerebral infarction (29%), nephrotic diseases (34%), and hypertension-related diseases (29%) had high false-positive rates, while diabetes (1.8%) and liver cirrhosis (1.1%) had high false-negative rates.


View this table:
[in this window]
[in a new window]
 
Table 3 Level of agreement in selecting underlying cause of death (UCD) between reviewer and original coders by main cause of death, in Taiwan, 1994a
 
Types of disagreements
According to the three-digit category, there were 1073 death certificates for which the UCD selected by the original coders was different from that selected by the reviewer. The types of disagreement were disagreement in nomenclature (42.8%), inappropriate judgement of causal relationship (41.5%), and incorrect interpretation of Selection Rule 3 and Modification Rules (15.7%).

Effects of disagreements on mortality statistics
The age-adjusted death rates (estimated according to the reviewer) and the death rate ratios (reviewer/original coders) were calculated for men (Table 4Go) and women (Table 5Go). The mortality rate ratio (MRR) is the net effect of false-positive and false-negative coding. Only for male liver cancer (MRR = 0.82, 95% CI : 0.67–0.99) and male nephrotic diseases (MRR = 0.69, 95% CI : 0.47–0.99) did the mortality rate estimated by the reviewer differ significantly from the mortality rate estimated by the original coder.


View this table:
[in this window]
[in a new window]
 
Table 4 Estimated male age-adjusteda death rates from main causes of death in Taiwan according to reviewer and original coders, in Taiwan, 1994b
 

View this table:
[in this window]
[in a new window]
 
Table 5 Estimated female age-adjusteda death rates from main causes of death in Taiwan according to reviewer and original coders, in Taiwan, 1994b
 

    Discussion
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
The results of this study indicate that the accuracy of cause-of-death statistics in Taiwan is not ideal. The level of agreement between the reviewer and the original coders in selection of the UCD varied according to the categories of codes (two- versus three-digit code), ICD-9 chapter, and main causes of death. Although overcoding and undercoding were prominent for some causes of death, owing to the ‘compensatory effect of errors’,22 the mortality statistics were not significantly affected.

The European Economic Community (EEC) studies showed great variations in coding practices among different countries.6–9 The investigators sent sets of case histories to samples of certifying physicians of different countries and the completed certificates were coded by national coding offices and by a WHO reference centre (as gold standard). The coding agreement rates between countries ranged from 60% to 92% in cases of COPD,6 80% to 94% in cases of cancer,7 and 29% to 98% in cases of diabetes.9 Our study showed poor coding agreement for liver cancer, cerebral infarction, nephrotic diseases, hypertension-related diseases, diabetes, and liver cirrhosis. There were several sources of disagreement, including choosing different codes, inappropriate judgement of causal relationship, and incorrect interpretation of Selection Rule 3 and Modification Rules.

Types and mechanisms of disagreement
The first type of disagreement involved the nomenclature of cause of death. Percy and Muir6 identified several sources of problems for coding death certificates mentioning cancer, including multiple sites of cancer, coexistent heart diseases and other cancers, interpretation of the coding rules, and translation problems. The translation problem is serious in Taiwan, because most medical doctors use English medical terms in clinical practice, while the death certificates are completed in Chinese. Different doctors might use different translations for the same English diagnosis. Because there is no Chinese version of the ICD-9 Index, the coders might also choose different codes. In fact, this was the most frequent type of disagreement between the reviewer and coders in our study (42.8%).

Another problem related to translation is that there are some controversial terms with diverse connotations. Although the term ‘neoplasm’ can denote either a malignant or a benign tumour depending on how it is further qualified, some certifiers interpret it as meaning a malignant tumour. The reviewer coded liver tumour as ICD-9 code 235 (neoplasm of uncertain behaviour of digestive and respiratory systems); however some original coders coded the same diagnosis as 155 (malignant neoplasm of liver and intrahepatic bile ducts), which caused the high false-positive rate for liver cancer.

The second type of disagreement was inappropriate judgement of causal relationships (41.5%). There was considerable divergence among the coders themselves and between some of the coders and the reviewer on the extent to which they were prepared to accept the statement of causation made by the certifying doctor. This divergence of approach is exemplified by one of the disputed death certificates, which read: I (a) Cerebral infarction (b) Diabetes mellitus (c) Hypertension. II Gastric cancer with metastases. The first coder might accept the certifier's statement of causation among different diseases, and selected hypertension as the UCD, according to the General Rule. The second coder might also accept the certifier's statement of causation, but used Rule 7 and selected cerebral infarction as the UCD. The third coder might judge that diabetes was ‘highly improbable’ as being due to hypertension, so this coder rejected the certifier's statement of causation and selected diabetes as the UCD according to Rule 1. The fourth coder might maintain that the metastasizing gastric cancer was the most serious condition, so this coder incorrectly interpreted Rule 3 and selected gastric cancer as the UCD. Thus, it appears that the judgement of the acceptance or rejection of the certifier's statement of causation is a key source of disagreement.

A study from Sweden showed that the coder's judgement and interpretation of the medical certificate play a decisive role in the data-collecting process for rheumatoid arthritis (RA) mortality statistics.13 In some situations the coders in the National Central Bureau of Statistics of Sweden had a preference for choosing RA as the UCD.14 As shown in Table 2Go, 46 cases in which the UCD selected by the reviewer was an endocrine disease (diabetes in most cases), the original coder determined a circulatory disease (most of them were CVA) to be the UCD. In these cases, the CVA was put on the first line and at the same time diabetes was on the last used line in Part I (of the cause-of-death section in the death certificate); in this situation, coders in Taiwan appear to prefer choosing CVA as the UCD.

A similar situation existed when liver cirrhosis and liver cancer were put on the same first line, with liver cirrhosis mentioned first: the reviewer chose liver cirrhosis as the UCD according to Rule 1, while most of the coders in Taiwan chose liver cancer. The coders' preferences for these diseases were according to the principle of severity, and this is the reason why diabetes and liver cirrhosis were undercoded. The false-negative rate for the two diagnoses were among the highest: 1.80% for diabetes and 1.09% for liver cirrhosis, respectively.

The third type of disagreement was incorrect interpretation of Rule 3 and the Modification Rules. As the EURODIAB study10,11 revealed, diabetes is a disease with certification and coding problems. Deciding whether diabetes is the cause of the death or a contributing factor is often difficult. Thus, diabetes may be undercoded in some situations and overcoded in others. If diabetes is put in Part II (of cause-of-death section in the death certificate) by the certifier, some coders in Taiwan might incorrectly use Rule 3 to select diabetes as the UCD. This is why there were 23 cases in which the reviewer selected CVD as the UCD while the original coder determined diabetes (in Part II) to be the UCD.

The interpretation of Rule 3 is always controversial. For example, in 1984 the Office of Population Censuses and Surveys introduced a revised interpretation of the WHO's Rule 3 in England and Wales.23 As a result, there was a large decrease in the number of reported deaths from causes such as pneumonia, while deaths from causes often mentioned in Part II increased in number.24 The automated cause coding system was implemented in England and Wales at the beginning of 1993, and the mortality rate from pneumonia reversed to the level of 1984.25 The change was caused by the fact that Rule 3 is interpreted differently in the US, where the automated system was designed.

Another example of the controversy in the interpretation of Rule 3 is the experience in Sweden, where the coders of the Central Bureau of Statistics routinely queried the death certificates with ‘pneumonia’ reported as the UCD. The Bureau conducted a retrospective study to abstract the medical records of decedents with pneumonia selected as the UCD on the death certificate. The study found that in about one-quarter of cases a condition mentioned in Part II was indeed the UCD; in another quarter of the cases the pneumonia was due to a serious disease that had not been mentioned on the certificate; and in the remaining half of the cases the pneumonia itself was judged to be the ‘true’ originating cause of death.26

Effects of disagreements on mortality statistics
Though the false-positive rates and false-negative rates were high for some diseases, the net effect of coding disagreements on most specific mortality rates was not significant in our study. This is possibly because of the ‘compensatory effect of errors', which was first proposed 40 years ago.22 For example, diabetes may be undercoded in some situations and overcoded in others, so the net effect was not significant in our study (MRR for diabetes was 0.96 for males and 1.02 for females).

The overcoding of cancer of the liver (MRR 0.82 for males and 0.75 for females by the original coders) can be further illustrated by comparing the incidence rates and mortality rates of this disease between Taiwan and the US. The incidence of hepatocellular carcinoma (HCC) in the US in 1994 was 4.0 per 100 000 for males and 1.4 per 100 000 for females; because the survival of HCC patients is poor, the mortality and incidence were virtually identical.27 Nevertheless, the mortality rate of HCC in Taiwan in 1993 (36.9 per 10 000 for males and 12.2 per 100 000 for females) was one-third higher than the incidence rate (25.6 per 100 000 for males and 8.7 per 100 000 for females),28 reflecting the overcoding of this disease in Taiwan.

The accuracy of coding for the general disease categories (e.g. the 10 leading causes of death) in Taiwan appears to be adequate, in that the ranking of these causes was not changed after review. Nevertheless, for diseases with small numbers of deaths (e.g. pregnancy-related deaths) the coding disagreements did have great impacts on mortality rates.

Limitations
Although we did not attempt to assess the accuracy of the diagnosis reported on the death certificates, errors in the way the doctor converts basic information about the patient into a diagnosis and completes the death certificate should also be noted as potential sources of error.29–35 Sources of error include the amount of information in cause-of-death diagnosis, the competence of the physician in deciding the main cause-of-death chain, and the wording in death certificate completion.

This study followed the same method used by previous investigations,4–11 using the UCD selected by the reviewer as the ‘gold standard’ for assessing the accuracy of coding. However, because selecting the UCD requires difficult interpretations of various selection rules, the process cannot be entirely objective. Nevertheless, in this study we not only showed the extent of the disagreements, but also the types and possible reasons for the disagreements. This information is critical for improving the cause-of-death coding system.

Implications for improvements
As we expected, the selection of a single UCD became more difficult in those who were older and had more morbid conditions. Some scholars have recommended the coding and publication of statistics on multiple causes of death to correct the problem of disagreement in selection of the UCD.36–39

The Head of the WHO Collaborating Center for Classification of Disease in Beijing has translated the ICD-10 index into Chinese. Nonetheless, the Chinese characters and medical terms used in Taiwan are quite different from those used in mainland China. Almost all physicians in Taiwan use English for writing medical records, and nosologists in hospitals use the English index of the ICD for coding. The Taiwan Department of Health is still evaluating the need for translation of the ICD-10 index into Taiwanese.

The agreement rates between the reviewer and each of the five coders were quite low compared with other studies.11–15 The Office of Statistics of the Provincial Government of Taiwan has now implemented routine double coding of samples of the death certificates completed by the five coders, to evaluate and improve the quality of the coding practices. Some of the coding problems discussed above could be resolved by using the ACME (Automated Classification of Medical Entities) computer program.18,25 The US National Center for Health Statistics sponsored a conference on an International Collaborative Effort on Automating Mortality Statistics, which was held in November 1996. A mortality coding forum on ICD-10 selection rules has been established on the Internet by Statistics Sweden (http://www.pubcare.uu.se/nordwho/verksam/mortfor.htm), to allow the coders in different countries to share their experiences and discuss coding-related questions. It is scheduled that ACME will be implemented when the ICD-10 is adopted in Taiwan.

In conclusion, the accuracy of cause-of-death coding in Taiwan appears to vary according to the type of disease. Types of disagreements between reviewer and the original coder included disagreement in nomenclature, inappropriate judgement of causal relationships, and incorrect interpretation of Selection Rule 3 and Modification Rules. Because of the ‘compensatory effect of errors’, the national mortality statistics were not affected significantly.


    Appendix
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
International rules for selection of underlying cause of death3
General rule. Select the condition entered alone on the lowest used line of Part I unless it is highly improbable that this condition could have given rise to all the conditions entered above it.

Rule 1. If there is a reported sequence terminating in the condition first entered on the certificate, select the underlying cause of this sequence. If there is more than one such sequence, select the underlying cause of the first-mentioned sequence.

Rule 2. If there is no reported sequence terminating in the condition first entered on the certificate, select this first mentioned condition.

Rule 3. If the condition selected by the General Rule or Rule 1 or 2 can be considered a direct sequel of another reported condition, whether in Part I or Part II, select this primary condition.

Modification of the underlying cause of death selected by the above rules

Rule 4. Senility.

Rule 5. Ill-defined conditions.

Rule 6. Trivial conditions.

Rule 7. Linkage.

Rule 8. Specificity.

Rule 9. Early and late stages of disease.

Rule 10. Late effects.

Rule 11. Old pneumonia, influenza and maternal conditions.

Rule 12. Errors and accidents.


    Acknowledgments
 
This study was supported by the Department of Health, Republic of China (DOH86-TD-017). The authors wish to thank Show-Ru Shau for assisting the statistical analysis and Jeff Radcliff for English editing of this manuscript.


    References
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Appendix
 References
 
1 Alderson MR. Mortality, Morbidity and Health Statistics. Basingstoke: Macmillan,1988, pp.105–67.

2 Start RD, Bury JB, Strachan AG, Cross SS, Underwood JCE. Evaluating the reliability of causes of death in published clinical research. Br Med J 1997;314:271.[Free Full Text]

3 World Health Organization. Manual of the International Statistical Classification of Diseases, Injuries, and Cause of Death, Vol 1. Geneva: WHO, 1977.

4 Anonymous. The accuracy and comparability of death statistics. WHO Chronicle 1967;21:11–17.[ISI][Medline]

5 Percy C, Dolman A. Comparison of the coding of death certificates related to cancer in seven countries. Public Health Rep 1978;93:335–50.[ISI][Medline]

6 Percy C, Muir C. The international comparability of cancer mortality data. Am J Epidemiol 1989;129:934–46.[Abstract]

7 Kelson MC, Heller R. The effect of death certification and coding practices on observed differences in respiratory disease mortality in 8 EEC countries. Rev Epidemiol Sante Publique 1983;31:423–32.[ISI][Medline]

8 Kelson M, Farebrother M. The effect of inaccuracies in death certification and coding practices in the European Economic Community (EEC) on international cancer mortality statistics. Int J Epidemiol 1987;16:411–14.[Abstract]

9 Mackenbach JP, Van Duyne WMJ, Kelson MC. Certification and coding of two underlying causes of death in the Netherlands and other countries of the European Community. J Epidemiol Community Health 1987;41:156–60.[Abstract]

10 Jougla E, Papoz L, Balkau B, Maguin P, Hatton F, the EURODIAB Subarea C Study Group. Death certificate coding practices related to diabetes in European Countries—The ‘EURODIAB Subarea C’ study. Int J Epidemiol 1992;21:343–51.[Abstract]

11 Balkau B, Jougla E, Papoz L, the EURODIAB Subarea C Study Group. European study of the certification and coding of causes of death of six clinical case histories of diabetic patients. Int J Epidemiol 1993; 22:116–26.[Abstract]

12 Curb JD, Babcock C, Pressel S, Tung B, Remington RD, Hawkins CM. Nosological coding of cause of death. Am J Epidemiol 1983;118: 122–28.[Abstract]

13 Lindahl BIB. The reliability of Swedish mortality statistics for rheumatoid arthritis. Scand J Rheumatol 1984;13:289–96.[ISI][Medline]

14 Lindahl BIB. In what sense is the rheumatoid arthritis the principal cause of death? A study of the National Statistics Office's way of reasoning based on 1224 death certificates. J Chron Dis 1985;38:963–72.[ISI][Medline]

15 Guibert RL, Wigle DT, Williams JI. Decline of acute myocardial infarction death rates not due to cause of death coding. Can J Public Health 1989;80:418–22.[ISI][Medline]

16 Baird PA, Sadovnick AD. Underlying causes of death in Down syndrome: accuracy of British Columbia death certificate data. Can J Public Health 1990;81:456–61.[ISI][Medline]

17 Carver JD, McDermott RJ, Jacobson HN et al. Infant mortality statistics do not adequately reflect the impact of short gestation. Pediatrics 1993;92:229–32.[Abstract]

18 National Center for Health Statistics. Instruction Manual Part 2C: ICD-9 ACME Decision Tables for Classifying Underlying Causes of Death. Hyattsville: US Department of Health and Human Services, Public Health Services, 1987.

19 Fleiss JL. Statistical Methods for Rates and Proportions. New York: Wiley, 1981, pp.212–25.

20 Ahlbom A. Biostatistics for Epidemiologists. Boca Raton: Lewis, 1993, pp.64–66.

21 World Health Organization. World Health Statistics 1993. Geneva: WHO, 1994.

22 James G, Patton RE, Heslin AS. Accuracy of cause-of-death statements on death certificates. Public Health Rep 1955;70:39–51.

23 Bain MRS, Chalmers JWT, Brewster DH. Routinely collected data in national and regional databases—an under-used resource. J Public Health Med 1997;19:413–18.[Abstract]

24 Martyn CN, Pippard EC. Usefulness of mortality data in determining the geography and time trends of dementia. J Epidemiol Community Health 1988;42:134–37.[Abstract]

25 Rooney C, Devis T. Mortality trends by cause of death in England and Wales 1980–94: the impact of introducing automated cause coding and related changes in 1993. Pop Trends 1997;86:29–35.

26 Johansson LA. Need for clarification of the ICD-10 mortality coding rules. Paper presented in the Meeting of Heads of WHO Collaborating Centers for the Classification of Diseases; 1996 October 15–21; Tokyo, Japan.

27 London WT, McGlynn KA. Liver cancer. In Schottenfeld D, Fraumeni JF (eds). Cancer Epidemiology and Prevention, 2nd edn. New York: Oxford University Press, 1996.

28 Department of Health. Cancer Registry Annual Report, Republic of China, 1993. Taipei: Department of Health, Taiwan, Republic of China, 1996.

29 Leadbeatter S. Semantics of death certification. J Royal Coll Phys London 1986;20:129–32.[ISI][Medline]

30 Zumwalt RE, Ritter MR. Incorrect death certification—an invitation to obfuscation. Postgrad Med 1987;81:245–54.[Medline]

31 Jordan JM, Bass MJ. Errors in death certificate completion in a teaching hospital. Clin Invest Med 1993;16:249–55.[ISI][Medline]

32 Messite J, Stellman SD. Accuracy of death certificate completion—the need for formalized physician training. JAMA 1996;275:794–96.[Abstract]

33 Maudsley G, Williams EMI. ‘Inaccuracy’ in death certification—where are we now? J Public Health Med 1996;18:59–66.[Abstract]

34 Magrane BP, Gilliland MGF, King DE. Certification of death by family physicians. Am Fam Physician 1997;51:1433–38.

35 Peach HG, Brumley DJ. Death certification by doctors in non-metropolitan Victoria. Aust Fam Physician 1998;27:178–82.[Medline]

36 Israel RA, Rosenberg HM, Curtin LR. Analytical potential for multiple cause-of-death data. Am J Epidemiol 1986;124:161–79.[ISI][Medline]

37 White MC, Selvin S, Merrill DW. A study of multiple causes of death in California: 1955 and 1980. J Clin Epidemiol 1989;42:355–65.[ISI][Medline]

38 Crew DE, Stamler J, Dyer A. Conditions other than underlying cause of death listed on death certificates provide additional useful information for epidemiological research. Epidemiology 1991;2:271–75.[Medline]

39 Westerling R. Small-area variation in multiple causes of death in Sweden—a comparison with underlying causes of death. Int J Epidemiol 1995;24:552–58.[Abstract]