Commentary: The prediction of coronary heart disease risk in individuals— an imprecise science

Peter Brindle and Margaret May

Department of Social Medicine, University of Bristol, Canynge Hall, Whiteladies Road, Bristol BS8 2PR, UK.

The paper by Thomsen and colleagues1 tells us that we should be cautious about applying a clinical decision rule developed in one population to another without first assessing its accuracy in that population. Unfortunately in the case of coronary risk prediction that is what has already happened. Coronary risk prediction using multiple risk factors has evolved as a method of helping clinicians prioritize prevention measures in patients who do not yet have overt cardiovascular disease. In the case of statins, restricting treatment to those individuals above an arbitrary level of risk, say 3% annual coronary heart disease (CHD) risk,2 has also provided a mechanism of limiting the costs of treating all those who might benefit (annual CHD risk >=0.6%) according to trial evidence.3 As treatment decisions are based on estimated levels of absolute risk, it is important that these estimates be as accurate as possible.

The Framingham risk equations4 and the clinical decision rules (CDR) that have been derived from them are proposed as the mainstay of primary prevention of CHD in Europe.5 There are three steps in the development of a CDR: deriving the rule, testing the rule, and assessing the impact of the rule on clinical behaviour.6 Until the publication of Thomsen’s paper, which is concerned with the second step of this CDR development process, and despite there being evidence that the Framingham risk equations overestimate risk in southern European countries,7 there has been no study that has adequately investigated the validity of any Framingham-based risk equation in a northern European country.

When testing the performance of a clinical prediction model in different populations, the accuracy of the predicted probability has two components (calibration and discrimination) which both need to be assessed.8 A well-calibrated model has predictions that are neither too high nor too low i.e. the baseline risk is correctly assessed. A model that discriminates well ranks individual risk in the correct order, i.e. it has high sensitivity and specificity. Using both validation criteria, the three studies that are usually cited as evidence of Framingham’s validity in northern European populations do not provide sufficiently robust evidence upon which the CHD primary prevention programme of Europe should be based.

In a theoretical modelling exercise, Haq et al. concluded that there was moderate agreement between Framingham and other northern European functions in the prediction of risk for individuals but they did not test whether any of the functions predicted observed events accurately.9 In an evaluation by the West of Scotland Coronary Prevention Study Group it was simply stated that the observed incidence of CHD events in the placebo arm of a statin trial ‘was close to that predicted by the Framingham regression function’.10 However, no numerical comparisons were supplied and no test of discrimination was performed. In contrast to the other two studies, Ramachandran et al. did use observed events in a non-trial population.11 They concluded that although there was no significant difference between the observed and predicted event rate in the higher risk population (>1.5% per year), Framingham underestimated the event rate in those at lower risk. Direct comparisons with Framingham were difficult as smoking was classified differently and for a rule designed to stratify individuals to different levels of risk, absence of any assessment of discriminatory ability left this attempt at validation lacking.

Thomsen and colleagues have been more complete in their assessment of the generalizability of a Framingham model. They have shown by using observed events, a Framingham model performs adequately compared with the Glostup model in the ranking of individuals and in the comparison of the relative risks of the well-known risk factors. However, the Framingham model consistently over-predicted risk in this Danish population because of the different baseline survival rates of the two populations.

The question that primary care practitioners in northern Europe want to know is ‘do our Framingham-based decision aids predict accurately in our practice population those individuals above and below a risk threshold for treatment?’ Thomsen and colleagues do not claim to answer this question, and they cannot do so for three main reasons. First, the Framingham data they used are different from those used in the derivation of the current risk scoring methods. Second, Thomsen defined ‘CHD death’ as the outcome and not ‘CHD events’ that is the endpoint used by the clinical prediction tools. Third, different statistical techniques were used in the original Framingham derivation and in the models developed for this validation exercise.

Since the baseline risk of CHD varies between countries there can be no single risk assessment equation that is valid throughout Europe. Even within a country important variation in CHD rates can occur.12 Furthermore, the incidence of CHD in parts of the developed world has declined considerably since the 1970s so any prediction method based on ‘old’ data will usually over-predict risk in an individual today. There is also additional uncertainty with patients who, unlike the Framingham study population are not white, are not between the ages of 30 and 74 or have a family history of CHD. Moreover, clinicians and patients are unaware of the precision of the risk estimates as no confidence intervals are given around the predicted risks.

The accuracy of the Framingham-based decision aids is further reduced because they are difficult to use in practice. McManus et al. found that only a fifth of patient records had all the information required to assess CHD risk and even when the information was complete, risk calculations made by general practitioners and practice nurses were only moderately accurate when compared to the ‘gold standard’ which the authors assumed to be the Framingham risk function.13 The authors concluded that adequate training is required to use these risk calculation rules, but even experts make mistakes. For example, authors of the British recommendations state that an absolute risk of non-fatal myocardial infarction or coronary death of >=30% over 10 years should be identified and treated with statins. However, they appeared to be unaware that the Framingham equations used in CDR predict a much wider outcome (i.e. coronary death, clinical non-fatal myocardial infarction, electrocardiographic myocardial infarction, physician assessed angina, and coronary insufficiency) which includes at least 50% more events, thereby increasing the absolute rate by the same amount.14 A correction has now been issued.15 Also recent guidance issued to all general practitioners in the UK on how to use the coronary risk prediction charts accurately16 stated that smokers who had quit in the last 5 years should be regarded as current smokers in the risk calculation. The definition of current smokers in the Framingham study, upon which these coronary risk prediction charts are based, includes those who have quit in the last year but not any earlier.17

Basing the primary prevention of CHD in individuals using coronary risk estimates from Framingham-derived risk estimates may not be as accurate or as easy as it is often thought to be. Despite the growing safety record and anticipated falling prices of statins, some appropriate measure of absolute risk will still remain useful to calculate the numbers needed to treat and for the evaluation of the cost-effectiveness of interventions. However, it is becoming clearer that a single risk function derived in one place at a particular time may not be applicable elsewhere. There is a need for more subtle and, perhaps, locally derived prediction rules which can be adjusted for geographical and temporal factors and are easy to apply in practice.

References

1 Thomsen TF, McGee D, Davidsen M, Jørgensen T. A cross-validation of risk-scores for CHD mortality based on data from the Glostrup Population Studies and Framingham Heart Study. Int J Epidemiol 2002;31:817–22.[Abstract/Free Full Text]

2 Department of Health. National Service Framework for Coronary Heart Disease. London: Department of Health, 2000.

3 Downs JR, Clearfield M, Weis S et al. Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. Air Force/Texas Coronary Atherosclerosis Prevention Study. JAMA 1998;279: 1615–22.[Abstract/Free Full Text]

4 Anderson KM, Odell PM, Wilson PW, Kannel WB. Cardiovascular disease risk profiles. Am Heart J 1991;121:293–98.[ISI][Medline]

5 Wood D, De Backer G, Faergeman O, Graham I, Mancia G, Pyorala K. Prevention of coronary heart disease in clinical practice: recommendations of the Second Joint Task Force of European and other Societies on Coronary Prevention. Atherosclerosis 1998;140: 199–270.[CrossRef][ISI][Medline]

6 McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS for the Evidence-Based Medicine Working Group. User’s Guides to the Medical Literature. XXII: How to use articles about clinical decision rules. JAMA 2000;284:79–84.[Abstract/Free Full Text]

7 Menotti A, Puddu PE, Lanti M. Comparison of the Framingham risk function-based coronary chart with risk function from an Italian population study. Eur Heart J 2000;21:365–70.[Abstract/Free Full Text]

8 Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999;130:515–24.[Abstract/Free Full Text]

9 Haq IU, Ramsay LE, Yeo WW, Jackson PR, Wallis EJ. Is the Framingham risk function valid for northern European populations? A comparison of methods for estimating absolute coronary risk in high risk men. Heart 1999;81:40–46.[Abstract/Free Full Text]

10 The West of Scotland Coronary Prevention Study Group. Baseline risk factors and their association with outcome in the West of Scotland Coronary Prevention Study. Am J Cardiol 1997;79:756–62.[CrossRef][ISI][Medline]

11 Ramachandran S, French JM, Vanderpump MP, Croft P, Neary RH. Using the Framingham model to predict heart disease in the United Kingdom: retrospective study. BMJ 2000;320:676–77.[Free Full Text]

12 Morris R, Whincup PH, Lampe F, Walker M, Wannamethee G, Shaper AG. Geographical variation of incidence of coronary heart disease in Britain: the contribution of established risk factors. J Epidemiol Community Health 2000;54:787–88.

13 McManus RJ, Mant J, Meulendijks CFM et al. Comparison of estimates and calculations of risk of coronary heart disease by doctors and nurses using different calculation tools in general practice: cross sectional study. BMJ 2002;324:459–64.[Abstract/Free Full Text]

14 Lampe FC, Walker M, Shaper AG, Brindle PM, Whincup PH, Ebrahim S. Endpoints for predicting coronary risk must be clarified. BMJ 2001;323:396.[Free Full Text]

15 Correction. Joint British recommendations on prevention of coronary heart disease in clinical practice: summary. BMJ 2001;323:780.[Free Full Text]

16 British Heart Foundation. How to use the coronary risk prediction charts for primary prevention. Factfile 01/2002.

17 Anderson KM, Wilson PWF, Odell PM, Kannel WB. An updated coronary risk profile—a statement for health professionals. Circulation 1991;83:356–62.[ISI][Medline]





This Article
Extract
FREE Full Text (PDF)
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Search for citing articles in:
ISI Web of Science (1)
Request Permissions
Google Scholar
Articles by Brindle, P.
Articles by May, M.
PubMed
PubMed Citation
Articles by Brindle, P.
Articles by May, M.