Department of Chemical Pathology, St. Thomas' Hospital, Lambeth Palace Road, London SE1 7EH, UK
The Ipswich Hospital, Ipswich, Suffolk IP4 5PD, UK
Queen's Hospital, Burton-on-Trent, Staffordshire DE13 0RB, UK
* Corresponding author. Tel.: +44-2079289292x2027; Fax: +44-2079284226
E-mail address: anthony.wierzbicki{at}kcl.ac.uk
Sir,
The paper by Empana et al.1 reviewing the utility of the Framingham and PROCAM cardiovascular risk calculation algorithms in a validation study against the PRIME cohort raises many issues. Firstly definitions are of critical importance for defining categorical variables.2 The definitions of diabetes and family history have either changed dramatically or are subject to wide differences in interpretation. This has profound effects on calculated risks. Given the results of this study, it is interesting to note that methods for measurement of HDL-cholesterol (HDL-C) have advanced since these algorithms were devised and modern assays give results 1020% higher than the methods used in Framingham resulting in an underestimation of risk due to low HDL-C in the original algorithm.
More fundamentally, all risk calculators have wide confidence intervals for predictions due to underlying biological variation in risk factors.3 We have previously shown that the 95% confidence interval at the accepted risk threshold is 20±6% for single measurement and 20±3.3% for triplicate measurements. These wide confidence intervals limit the power of any algorithm to reliably identify high-risk individuals as is clearly demonstrated in this paper.
This study uses the 1998 version of the Framingham algorithm, which relies on multiple sub-categories of LDL-cholesterol (LDL-C) to fine tune risk estimates. In the UK, as in many other countries and commercial computer programs, the original simpler 1991 algorithm is used. Given the high biological, analytical and mathematical variation in calculated LDL-C, we would contend that this more modern version may be less accurate than the older algorithm. The basic problem lies in the high biological variability of triglycerides and their consequent effect on the Friedwald equation a foundation stone of modern cardiology anchored on the shifting sands of a small number of patients.4 Direct measurement of LDL-C, though likely more accurate, is rarely performed and is subject to methodological differences between assays.
Despite these limitations, both risk calculators did predict high-risk groups but over-estimated the likely burden of disease. However, only proportions of events predicted were assessed. When attempting to predict risk of an event in an individual using a population-based function, identical proportions from two assessment systems may identify profoundly different populations. This is the problem of concordance. Concordance between events predicted in high-risk individuals and actual individual outcomes was not assessed in this study and may represent a further source of error in risk calculation. Indeed even minor modifications to risk assessment programs can result in significant changes in concordance.5 Similarly evolution of risk factor distributions (e.g., increase in diabetes) is likely to cause drift in algorithm coefficients (as is clearly shown in Table 2 in the paper) leading to further reductions in concordance between predicted and actual events.
Given these limitations we would contend that the interpretation of cardiovascular risk calculation algorithms is an art. Clinicians should avoid excessive reliance on spuriously accurate computer predictions but should instead critically review the individual risk factors in each patient and base their judgements on initiation of intervention on a broad view of the likely risks.
References