Department of Social Medicine, University of Bristol, Canynge Hall, Whiteladies Road, Bristol BS8 2PR, UK.
The paper by Thomsen and colleagues1 tells us that we should be cautious about applying a clinical decision rule developed in one population to another without first assessing its accuracy in that population. Unfortunately in the case of coronary risk prediction that is what has already happened. Coronary risk prediction using multiple risk factors has evolved as a method of helping clinicians prioritize prevention measures in patients who do not yet have overt cardiovascular disease. In the case of statins, restricting treatment to those individuals above an arbitrary level of risk, say 3% annual coronary heart disease (CHD) risk,2 has also provided a mechanism of limiting the costs of treating all those who might benefit (annual CHD risk 0.6%) according to trial evidence.3 As treatment decisions are based on estimated levels of absolute risk, it is important that these estimates be as accurate as possible.
The Framingham risk equations4 and the clinical decision rules (CDR) that have been derived from them are proposed as the mainstay of primary prevention of CHD in Europe.5 There are three steps in the development of a CDR: deriving the rule, testing the rule, and assessing the impact of the rule on clinical behaviour.6 Until the publication of Thomsens paper, which is concerned with the second step of this CDR development process, and despite there being evidence that the Framingham risk equations overestimate risk in southern European countries,7 there has been no study that has adequately investigated the validity of any Framingham-based risk equation in a northern European country.
When testing the performance of a clinical prediction model in different populations, the accuracy of the predicted probability has two components (calibration and discrimination) which both need to be assessed.8 A well-calibrated model has predictions that are neither too high nor too low i.e. the baseline risk is correctly assessed. A model that discriminates well ranks individual risk in the correct order, i.e. it has high sensitivity and specificity. Using both validation criteria, the three studies that are usually cited as evidence of Framinghams validity in northern European populations do not provide sufficiently robust evidence upon which the CHD primary prevention programme of Europe should be based.
In a theoretical modelling exercise, Haq et al. concluded that there was moderate agreement between Framingham and other northern European functions in the prediction of risk for individuals but they did not test whether any of the functions predicted observed events accurately.9 In an evaluation by the West of Scotland Coronary Prevention Study Group it was simply stated that the observed incidence of CHD events in the placebo arm of a statin trial was close to that predicted by the Framingham regression function.10 However, no numerical comparisons were supplied and no test of discrimination was performed. In contrast to the other two studies, Ramachandran et al. did use observed events in a non-trial population.11 They concluded that although there was no significant difference between the observed and predicted event rate in the higher risk population (>1.5% per year), Framingham underestimated the event rate in those at lower risk. Direct comparisons with Framingham were difficult as smoking was classified differently and for a rule designed to stratify individuals to different levels of risk, absence of any assessment of discriminatory ability left this attempt at validation lacking.
Thomsen and colleagues have been more complete in their assessment of the generalizability of a Framingham model. They have shown by using observed events, a Framingham model performs adequately compared with the Glostup model in the ranking of individuals and in the comparison of the relative risks of the well-known risk factors. However, the Framingham model consistently over-predicted risk in this Danish population because of the different baseline survival rates of the two populations.
The question that primary care practitioners in northern Europe want to know is do our Framingham-based decision aids predict accurately in our practice population those individuals above and below a risk threshold for treatment? Thomsen and colleagues do not claim to answer this question, and they cannot do so for three main reasons. First, the Framingham data they used are different from those used in the derivation of the current risk scoring methods. Second, Thomsen defined CHD death as the outcome and not CHD events that is the endpoint used by the clinical prediction tools. Third, different statistical techniques were used in the original Framingham derivation and in the models developed for this validation exercise.
Since the baseline risk of CHD varies between countries there can be no single risk assessment equation that is valid throughout Europe. Even within a country important variation in CHD rates can occur.12 Furthermore, the incidence of CHD in parts of the developed world has declined considerably since the 1970s so any prediction method based on old data will usually over-predict risk in an individual today. There is also additional uncertainty with patients who, unlike the Framingham study population are not white, are not between the ages of 30 and 74 or have a family history of CHD. Moreover, clinicians and patients are unaware of the precision of the risk estimates as no confidence intervals are given around the predicted risks.
The accuracy of the Framingham-based decision aids is further reduced because they are difficult to use in practice. McManus et al. found that only a fifth of patient records had all the information required to assess CHD risk and even when the information was complete, risk calculations made by general practitioners and practice nurses were only moderately accurate when compared to the gold standard which the authors assumed to be the Framingham risk function.13 The authors concluded that adequate training is required to use these risk calculation rules, but even experts make mistakes. For example, authors of the British recommendations state that an absolute risk of non-fatal myocardial infarction or coronary death of 30% over 10 years should be identified and treated with statins. However, they appeared to be unaware that the Framingham equations used in CDR predict a much wider outcome (i.e. coronary death, clinical non-fatal myocardial infarction, electrocardiographic myocardial infarction, physician assessed angina, and coronary insufficiency) which includes at least 50% more events, thereby increasing the absolute rate by the same amount.14 A correction has now been issued.15 Also recent guidance issued to all general practitioners in the UK on how to use the coronary risk prediction charts accurately16 stated that smokers who had quit in the last 5 years should be regarded as current smokers in the risk calculation. The definition of current smokers in the Framingham study, upon which these coronary risk prediction charts are based, includes those who have quit in the last year but not any earlier.17
Basing the primary prevention of CHD in individuals using coronary risk estimates from Framingham-derived risk estimates may not be as accurate or as easy as it is often thought to be. Despite the growing safety record and anticipated falling prices of statins, some appropriate measure of absolute risk will still remain useful to calculate the numbers needed to treat and for the evaluation of the cost-effectiveness of interventions. However, it is becoming clearer that a single risk function derived in one place at a particular time may not be applicable elsewhere. There is a need for more subtle and, perhaps, locally derived prediction rules which can be adjusted for geographical and temporal factors and are easy to apply in practice.
References
1 Thomsen TF, McGee D, Davidsen M, Jørgensen T. A cross-validation of risk-scores for CHD mortality based on data from the Glostrup Population Studies and Framingham Heart Study. Int J Epidemiol 2002;31:81722.
2 Department of Health. National Service Framework for Coronary Heart Disease. London: Department of Health, 2000.
3 Downs JR, Clearfield M, Weis S et al. Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. Air Force/Texas Coronary Atherosclerosis Prevention Study. JAMA 1998;279: 161522.
4 Anderson KM, Odell PM, Wilson PW, Kannel WB. Cardiovascular disease risk profiles. Am Heart J 1991;121:29398.[ISI][Medline]
5 Wood D, De Backer G, Faergeman O, Graham I, Mancia G, Pyorala K. Prevention of coronary heart disease in clinical practice: recommendations of the Second Joint Task Force of European and other Societies on Coronary Prevention. Atherosclerosis 1998;140: 199270.[CrossRef][ISI][Medline]
6 McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS for the Evidence-Based Medicine Working Group. Users Guides to the Medical Literature. XXII: How to use articles about clinical decision rules. JAMA 2000;284:7984.
7 Menotti A, Puddu PE, Lanti M. Comparison of the Framingham risk function-based coronary chart with risk function from an Italian population study. Eur Heart J 2000;21:36570.
8 Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999;130:51524.
9 Haq IU, Ramsay LE, Yeo WW, Jackson PR, Wallis EJ. Is the Framingham risk function valid for northern European populations? A comparison of methods for estimating absolute coronary risk in high risk men. Heart 1999;81:4046.
10 The West of Scotland Coronary Prevention Study Group. Baseline risk factors and their association with outcome in the West of Scotland Coronary Prevention Study. Am J Cardiol 1997;79:75662.[CrossRef][ISI][Medline]
11 Ramachandran S, French JM, Vanderpump MP, Croft P, Neary RH. Using the Framingham model to predict heart disease in the United Kingdom: retrospective study. BMJ 2000;320:67677.
12 Morris R, Whincup PH, Lampe F, Walker M, Wannamethee G, Shaper AG. Geographical variation of incidence of coronary heart disease in Britain: the contribution of established risk factors. J Epidemiol Community Health 2000;54:78788.
13 McManus RJ, Mant J, Meulendijks CFM et al. Comparison of estimates and calculations of risk of coronary heart disease by doctors and nurses using different calculation tools in general practice: cross sectional study. BMJ 2002;324:45964.
14 Lampe FC, Walker M, Shaper AG, Brindle PM, Whincup PH, Ebrahim S. Endpoints for predicting coronary risk must be clarified. BMJ 2001;323:396.
15 Correction. Joint British recommendations on prevention of coronary heart disease in clinical practice: summary. BMJ 2001;323:780.
16 British Heart Foundation. How to use the coronary risk prediction charts for primary prevention. Factfile 01/2002.
17 Anderson KM, Wilson PWF, Odell PM, Kannel WB. An updated coronary risk profilea statement for health professionals. Circulation 1991;83:35662.[ISI][Medline]