Introduction

Heart failure (HF) affects >5 million Americans.¹ Unless effective preventive interventions are implemented, it is estimated that by 2030, HF prevalence will grow by 25% and annual costs of care from $21 to $53 billion.² The prognosis of patients with HF remains poor with a projected 50% 5-year mortality, underscoring the importance of prevention.³ Prevention interventions, however, are unlikely to be cost-effective unless targeted toward reliably identified high-risk populations. Because of the epidemiological paradox, more HF cases occur in patients with the absence of any given individual risk factor, which makes it important to develop a more comprehensive risk assessment scheme than to rely solely on individual risk factors. Unfortunately, the best approach for quantifying overall incident HF risk in individuals remains unclear. In this study, we sought to systematically assess the performance characteristics and the use of the available HF risk models for predicting incident HF in the general population.

Clinical Perspective on p 447

Methods

Data Sources

We searched PubMed and EMBASE databases from January 1990 to August 2014 for HF risk prediction models published in English language. We used search terms related to HF and disease prediction (Appendix A in the Data Supplement) and reviewed the reference lists of studies and identified their citations through the ISI Web of Science for additional data. Eligible studies had to report a risk assessment model in a cohort study either by an equation or by a score. Two evaluators (J.B.E. and J.B.) independently identified articles and screened them for the inclusion criteria (Figure) by reviewing text and the Data Supplement. Disagreements were solved by consensus. We excluded studies for models developed in specific population (eg, postoperative patients, patients already diagnosed with HF, studies that reported measures of risk without information on model characteristics, and simulation studies). We extracted information on the area under the receiver-operating characteristic curve (AUC) or c-statistic, reclassification percentage, net reclassification improvement (NRI), and integrated discrimination improvement index (IDI).⁴

• Download figure
• Download PowerPoint

Bias, Clinical Usefulness, and External Validity

Risk for bias in models was assessed per Hayden et al⁵ recommendations, including 7 categories, such as participation (sampling bias), attrition (attrition bias), prognostic factor selection, prognostic factor measurement, outcome measurement (ascertainment bias), statistical analysis, and reporting model performance. Extent and methods of dealing with missing data, (Table in the Data Supplement), appropriateness of statistical methods, and clinical usefulness as a combination of clinical use and usability were assessed. Clinical use was assessed as whether predictive models proposed clinical scenarios or decision nodes linked to a risk category or threshold where the model may be useful for guiding diagnostic or therapeutic interventions. For usability, we noted whether a calculator or nomogram that enables easier knowledge translation was added. To examine generalizability and external validity, we evaluated whether models had been validated in an independent population in the original or a subsequent publication.

Data Extraction and Quality Assessment

Two reviewers (J.B.E. and J.B.) performed quality assessment and extracted information about design, setting, population characteristics, number of patients in the derivation and validation cohorts, number of outcomes of interest, number of candidate predictor variables, and the type of regression model used. For predictors, we specifically assessed for age, sex, hypertension, left ventricular hypertrophy, myocardial infarction, diabetes mellitus, valvular disease, and overweight/obesity, based on previously reported association of these factors with HF. For discriminative ability, information on the c-statistic, and for calibration, the difference between the observed and predicted rates, as well as the P value of the corresponding test statistic, were extracted. Data relevant to reclassification included NRI and IDI values, as well as their respective 95% confidence intervals and P values, when available. Given the heterogeneity in the type and number of metrics, the number and nature of risk factors included, and the study designs used, our analysis consisted mainly of a narrative synthesis of the evidence.

Effect of HF Risk Models

Effect of studies was identified through scanning the references of all publications and applying search strategies for impact studies proposed by Reilly and Evans,⁶ which combines the model’s acronym or name of the cohort or first author with specific search term combination (Appendix B in the Data Supplement). We searched relevant practice guidelines to assess the implementation of HF prediction models in countries or regions, in which such models were developed. We also targeted English-language guidelines compiled by a selection of professional organizations known to regularly issue recommendations for the management of HF, including the American College of Cardiology,⁷ the American Heart Association,⁷ Heart Failure Society of America,⁸ and the European Society of Cardiology.⁹

Results

Prediction Models

We screened 413 abstracts, retrieved 48 citations for full-text review, and selected 13 studies that described 28 HF prediction risk models (Figure). The characteristics of the risk prediction models are presented in Table 1.

Populations, Risk Factors, and Outcomes

Eleven models were developed from US cohorts, 1 in Sweden, 1 in Britain, and 2 multicenter cohorts (19 European countries in 1 study¹⁷ and 38 countries from America, Europe, Asia, and Africa in another study²²). Models were mostly developed in whites; the models based on US populations included non-white participants, mainly blacks. The number of participants ranged from 725 to 359 947 and 18 to 99 years of age. The length of follow-up ranged from 2.9 to 14 years. Three models were developed in specific populations, including diabetes mellitus,¹⁷ impaired glucose tolerance,²² and atrial fibrillation.²¹ The method of HF identification varied, but was mainly based on clinical presentation, hospital discharge records, and cause of death. One study defined HF using outpatient records in addition to hospital and death record¹² and another study included an echocardiographic-based definition of HF.¹⁸ None of the studies assessed HF with preserved ejection fraction and HF with reduced ejection fraction separately.

One study published the numbers of variables tested for inclusion.²² Thus for most studies, the number of outcomes of interest per candidate variable could not be assessed. The predictors most commonly included in the final models were age, sex, body mass index, diabetes mellitus, systolic blood pressure, creatinine, serum albumin or total protein, LV hypertrophy, and coronary artery disease. Biomarkers were evaluated in several studies, including B-type natriuretic peptide (BNP) or N-terminal pro-BNP (NT-proBNP), midregional proatrial natriuretic peptide, interleukin (IL)-6), tumor necrosis factor (TNF)-α, C-reactive protein (CRP), soluble IL-2 receptor, growth differentiation factor-15, high-sensitivity troponin I (hsTnI), and urinary albumin/creatinine ratio.

All the models were developed using a Cox regression model with reporting of the model with β coefficients, with 2 studies providing additional point-based scoring systems.^10,11

Performance of the Models

Table 1 displays the model performance. The AUC ranged from 0.71 to 0.87, indicating a modest-to-good discrimination. Risk scores were internally validated through bootstrapping in 4^11,17,20,22 and jackknife in 1 study.¹³ Five models had calibration estimated with the Hosmer–Lemeshow test. Most studies did not test for sex-based^{10–13,16–18,21,22} or race-based^{10–13,15–19,21,22} specific performance, except the Health Aging, and Body Composition (ABC) study HF score that predicted HF risk similarly in both sex and white/black race in development cohort,¹¹ but the risk was underestimated in white men, especially in the 5% to 10% risk category, in the validation cohort.²⁴ The AUCs were 0.71, 0.75, 0.74, and 0.72 in white men, white women, black men, and black women, respectively.²⁴

Model Improvement

In the Framingham Study, BNP and the urinary albumin/creatinine ratio increased the AUC from 0.84 to 0.86¹³ and 13% of subjects were appropriately reclassified. In the Health ABC study, the addition of IL-6, TNF-α, and CRP increased AUC from 0.717 to 0.737, mainly because of IL-6 that improved AUC to 0.734 (P=0.001).¹⁶ In the Cardiovascular Health Study, NT-proBNP increased AUC from 0.667 to 0.719 (P<0.001), but discrimination remained moderate.²⁵ In the Malmö Diet and Cancer study, midregional proatrial natriuretic peptide, NT-proBNP, and CRP together improved AUC from 0.815 to 0.842 with an IDI of 0.05 (P<0.001) and an NRI of 22% (P<0.001).¹⁵ In the Atherosclerosis Risk in Communities study, adding NT-proBNP increased the AUC by 0.032, NRI by 13%, and IDI by 0.057; NRI was higher in women (17.9%) than in men (14.4%).²⁰

Echocardiographic score (left ventricular ejection fraction and mass, E/A ratio, and left atrium), and NT-proBNP improved AUC (echocardiography, 0.031; NT-proBNP, 0.027; combined, 0.043; all P<0.01), and yielded robust IDI (echocardiography, 43.3%; NT-proBNP, 42.2%; combined, 61.7%; all P<0.001), and NRI (echocardiography, 11.3%; NT-proBNP, 10.6%; combined, 16.3%; all P<0.01), when added to the Health ABC model.¹⁸ Participants at intermediate risk (5%–20% 5-year risk; 35.7% of the cohort) derived the most reclassification benefit. Echocardiography yielded modest reclassification when used sequentially after NT-proBNP.

A multimarker score with soluble ST2, growth differentiation factor-15, hsTnI, hsCRP, and BNP increased AUC (0.846–0.870), NRI (0.39), and IDI (0.04) in the Framingham study.¹⁹ In the Atherosclerosis Risk in Communities study, cardiac troponin T and NT-proBNP improved AUC by 0.040 and 0.057; and NRIs by 50.7% and 54.7% in women and in men, respectively.²⁶ In the Multi-Ethnic Study of Atherosclerosis cohort, NT-proBNP and LV mass index change AUC from 0.81 to 0.89; IDI by 0.046; NRI 6-year risk probability categorized by <3%, 3% to 10%, >10% of 0.175, P=0.019 and category-less NRI by 561, P<0.001.²⁷ In the Framingham cohort, a composite of LV hypertrophy and LV dysfunction increased AUC from 0.765 to 0.770. With growth differentiation factor-15 and hsTnI, the NRI was 0.212 (P<0.0001).²⁸ In the British Heart Regional Study, the addition of NT-proBNP to the Health ABC model improved AUC from 0.706 to 0.764 (P<0.0001).²³ In the Prevention of Vascular and Renal End stage Disease (PREVEND) study, NT-proBNP, midregional proatrial natriuretic peptide, hsTnT, hsCRP, cystatin-C and urinary albumin excretion individually modestly improved the AUC of 0.826 by 4.7, 2.2, 2.5, 1.9, 1.0, and 1.6%, respectively.²⁹

External Validity

Five models were externally validated (Table 2). Of these, the Framingham¹⁰ and Health ABC models¹¹ were externally validated more than once. The AUC in validation studies were generally lower and with a poorer calibration, although the latter was not assessed in most of validation studies.

Single Biomarker or Imaging Data

In the Cardiovascular Health Study, a comparison of the Health ABC score, NT-proBNP, and echocardiographic data for detecting LV dysfunction indicated >5% risk of HF at 5 years in 44% of the cohort and identified 76% of those who developed HF by 5 years.^24,25 An abnormal NT-proBNP was present in 29.5% of the cohort and identified 45% of those who developed HF by 10 years; and baseline echocardiography identified reduced ejection fraction in 7.2% of the population, but identified only 12% of those who developed HF by 10 years.

Depending on the method of indexing LV mass (body surface area, fat-free mass, or height), the AUC for the Health ABC HF score (0.73, 0.75, and 0.64, respectively) was greater than that for BNP (0.62, 0.70, and 0.57, respectively) and NT-pro-BNP (0.62, 0.69, and 0.56, respectively; P<0.01 for all). Similar results were seen in the Framingham score except when LV mass was indexed to fat-free mass (0.65, 0.69 and 0.66, and 0.64, respectively). Addition of BNP resulted in an improvement in discrimination of stage B HF (difference in AUC, 0.01–0.03; P<0.05 for all).³⁰ In an elderly at high risk, NT-proBNP (AUC, 0.74) was superior to the Framingham HF score (AUC, 0.63) in predicting asymptomatic LV systolic dysfunction.³¹

Model Effect

There were no recommendations for using any risk prediction model to estimate the risk of incident HF in either the clinical or the community settings in any of the existing guidelines. There were no studies assessing the effect of adopting HF risk models in clinical practice on the process of care and outcomes of patients.

Discussion

This systematic review demonstrates the feasibility of global HF risk prediction in the general population. Compared with coronary artery disease and stroke, prediction of incident HF risk is at an early stage and is not incorporated into contemporary clinical or public health practice. Defining risk can be reliably done using a combination of common variables with an acceptable-to-good discrimination, although correcting for overfitting (internal validation) or testing in a new population (external validation) may attenuate discrimination. Implementation of prediction models may identify a wider segment of at-risk population compared with single risk factors. In addition, biomarkers may be used to further categorize patients into different risk strata in concert with the clinical variables.

HF guidelines advocate prevention of progression from at-risk HF stages (A and B) to symptomatic stages (C and D), thus requiring identification of susceptible patients likely to benefit from interventions,⁷ including lifestyle changes or more intense risk factor modification. Risk communication to patients may motivate adherence; indeed healthy lifestyle is associated with a lower lifetime risk of HF.³² Furthermore, risk scores can be incorporated into electronic medical records, allowing continual passive screening of high-risk subjects with alerts for patients entering higher-risk strata. These risk models may also circumvent uncertainties surrounding the stage B HF definition, which is currently dictated by what can be measured with imaging rather than a complete understanding of risk. In stepwise screening with models as the first step followed by imaging or biomarkers, those with stage B HF may be more reliably referred for interventions known to halt and possibly reverse structural changes that precede clinical HF. With increasing HF risk, LV mass and concentric remodeling increases, supporting the traditional belief that HF risk correlates with structural heart disease.³⁰ HF models may be useful in assessing the use of novel biomarkers for risk prediction or for selecting patients for inclusion in clinical trials. Indeed, investigating novel markers is warranted, especially given the relatively modest additional discriminative ability of natriuretic peptides (BNP or NT-proBNP) beyond that of clinical variables. This finding is not surprising because factors included in models (eg, plasma creatinine, female sex, and age) may increase these biomarkers independently of left ventricular ejection fraction and filling pressures.³³ Moreover, several screening studies have found use of natriuretic peptide level as stand-alone test to lack discriminative ability for detecting structural abnormalities among healthy young adults.^34,35

Some of the existing models have been statistically underpowered, especially as consideration was not given to the ratio of outcomes to predictors. Models with an inappropriate ratio (<10 events per variable) are less robust, with a performance that shrinks when applied to other populations. Although the overall study size and the prognostic ability of individual predictors is important, a key driver of statistical power is the number of events. These performance metrics mostly originated from the derivation sample, which are optimistic. Some models were internally validated (bootstrap or jackknife validation), providing an idea about their potential performance when applied to a new population. Calibration was not always assessed. A poorly calibrated model may have the same discrimination as a well-calibrated model, necessitating both of these characteristics to be assessed in concert. Attempts to update models by extra variables have focused on discrimination,^{13,15,16,18–20,25} and none of the studies reported change in the calibration. Finally, validation is an important step before a model can be recommended for widespread use. This was limited to only 3 models in US populations. Hence, further validations of existing models are needed to guarantee their generalizability.

HF models have mainly been published as mathematical equations. Complex formulas may not be suitable for clinical application. Indeed, previous research suggests that many clinicians find risk calculation too time consuming and are not convinced of the value of information derived from these models.³⁶ Additional translation efforts are needed to convert predictions equations into user-friendly tools to improve uptake.¹⁵ Efforts are needed to derive the appropriate context for defining high-risk status to facilitate their use. It is also important to study whether implementing HF risk prediction affects the behavior of healthcare providers and improves clinical outcomes. Few studies examined the incremental predictive value of novel biomarkers (eg, collagen matrix^37,38 and omic markers) that may be useful for risk stratification. Clinical risk models provide good basis to assess the incremental value of new discoveries.

Further limitations to existing models exist. Misclassification of HF events may have occurred as there are no universally agreed specific criteria for diagnosing HF, thus affecting the performance of the models. HF is a heterogeneous syndrome, representing a spectrum of pathogenesis, as well as normal to depressed ejection fraction. Existing studies use multiple criteria to define HF³⁹ with little agreement.⁴⁰ Inconsistencies were indeed observed in the presently reviewed studies, which may have affected model performance. In most studies, diagnosis did not rely on imaging. Consequently, specific forms of HF, such as HF with reduced ejection fraction and HF with preserved ejection fraction, were not considered. Although there is an overlap between predictors of HF with reduced ejection fraction and HF with preserved ejection fraction, there are differences that merit consideration.^41,42 Moreover, HF with reduced ejection fraction and HF with preserved ejection fraction are regarded as distinct pathophysiologic entities, as exemplified by the lack of evidence-based disease modifying therapies for the later. Accordingly, it stands to reason that differing preventative strategies and interventions may exist. Furthermore, the majority of participants included were whites. However, HF risk is known to be heightened in specific groups (eg, blacks), thus supporting the need to incorporate more patients of different ethnic backgrounds.

Although existing models may lead to implementation of prevention interventions, such efforts may be premature, particularly in the absence of external validation studies and impact analysis. The lack of studies evaluating the effect of model use on outcomes should be addressed in future studies. Such an undertaking could potentially strengthen the case for their use in routine clinical or public health practice. The absence of existing studies may simply be an indication of the embryonic nature of the HF prediction field, and of HF prevention in general. This limitation is not specific to HF and does not in itself invalidate the future use of the models. Indeed, even with coronary heart disease, for which prediction models have been in existence because the early 1990s, only a limited number of studies have assessed the effect of their implementation, although their use has been consistently recommended in guidelines. We did not explicitly rank or categorize the quality of existing HF models, as there is no agreed-on scientific system for rating risk prediction model quality. Some argue minimizing risk for potential bias is of critical importance, whereas others contend that a risk score should be judged on ability to perform accurately across diverse settings. Finally, our ability to assess publication bias was limited.

In conclusion, risk models for predicting incident HF have acceptable-to-good discriminative ability. These tools need to be better calibrated and externally validated, and the effect of their use on outcomes assessed before incorporation into clinical practice guidelines. With advancement of our understanding of systemic conditions and cardiac and noncardiac structural and functional abnormalities that predispose to HF risk, better approaches for HF prevention are needed. HF prevention extends beyond simply addressing risk factors or precursors, such as diabetes mellitus, hypertension, and coronary artery disease, especially as the latter is not a sine qua non for the development of HF. HF scores may be viewed as a means for selecting a targeted population that would undergo imaging for detection of asymptomatic structural disease (eg, left ventricular systolic dysfunction), a potential therapeutic target in itself. Successful prevention approaches mandate the correct identification of the target population, underscoring the importance of further research in the performance and use of incident HF risk models.

Footnotes