Cardiovascular Risk Prediction Functions Underestimate Risk in HIV Infection
Cardiovascular disease (CVD) risk is elevated in HIV-infected individuals, with contributions from both traditional and nontraditional risk factors. The accuracy of established CVD risk prediction functions in HIV is uncertain. We sought to assess the performance of 3 established CVD risk prediction functions in a longitudinal cohort of HIV-infected men.
The FHS (Framingham Heart Study) functions for hard coronary heart disease (FHS CHD) and atherosclerotic CVD (FHS ASCVD) and the American College of Cardiology/American Heart Association ASCVD function were applied to the Partners HIV cohort. Risk scores were calculated between January 1, 2006, and December 31, 2008. Outcomes included CHD (myocardial infarction or coronary death) for the FHS CHD function and ASCVD (myocardial infarction, stroke, or coronary death) for the FHS ASCVD and American College of Cardiology/American Heart Association ASCVD functions. We investigated the accuracy of CVD risk prediction for each function when applied to the HIV cohort using comparison of Cox regression coefficients, discrimination, and calibration.
The HIV cohort was comprised of 1272 men followed for a median of 4.4 years. There were 78 (6.1%) ASCVD events; the 5-year incidence rate was 16.4 per 1000 person-years. Discrimination was moderate to poor as indicated by the low c statistic (0.68 for FHS CHD, 0.65 for American College of Cardiology/American Heart Association ASCVD, and 0.67 for FHS ASCVD). Observed CVD risk exceeded the predicted risk for each of the functions in most deciles of predicted risk. Calibration, or goodness of fit of the models, was consistently poor, with significant χ2P values for all functions. Recalibration did not significantly improve model fit.
Cardiovascular risk prediction functions developed for use in the general population are inaccurate in HIV infection and systematically underestimate risk in a cohort of HIV-infected men. Development of tailored CVD risk prediction functions incorporating traditional CVD risk factors and HIV-specific factors is likely to result in more accurate risk estimation to guide preventative CVD care.
Editorial, see p 2215
What Is New?
In this study evaluating cardiovascular disease (CVD) risk prediction in HIV populations, established CVD risk prediction functions were shown to underestimate risk in HIV-infected men and poorly discriminate between individuals who experience an outcome from those who do not.
The study adds new knowledge by evaluating several risk functions in parallel for the first time, including the FHS (Framingham Heart Study) function for hard coronary heart disease and 2 functions that predict atherosclerotic CVD, the American College of Cardiology/American Heart Association function and an adapted FHS global CVD function.
What Are the Clinical Implications?
Our findings suggest that established CVD risk functions may not accurately estimate risk in the setting of HIV disease and may fail to identify men at elevated CVD risk who would benefit from aggressive risk reduction.
The incorporation of HIV-specific variables into current CVD risk prediction algorithms may be necessary to accurately predict risk for this population that is already at heightened risk.
Tailored CVD risk prediction strategies may apply to other high-risk populations with chronic inflammatory conditions.
Cardiovascular disease (CVD) risk prediction functions are widely used to predict CVD risk and prevent disease through identification of the highest-risk individuals who merit intensive risk factor modification. HIV-infected individuals face an increased CVD risk, with multiple studies demonstrating a 1.5- to 2-fold increased risk of myocardial infarction (MI) or stroke compared with non-HIV–infected individuals.1–3 Although traditional CVD risk factors such as smoking, dyslipidemia, diabetes mellitus, and hypertension are prevalent in HIV,2,4–10 emerging data support a prominent role of novel CVD risk factors related to chronic inflammation and immune activation in HIV-related CVD risk.11–15 Moreover, specific HIV medications have been shown to alter CVD risk.16,17 Established CVD risk functions were developed using risk factors common in the general population and do not reflect these unique mechanistic factors thought to drive HIV-related CVD. Consequently, existing functions may not accurately estimate risk in the HIV setting.18
The most widely applied CVD risk functions19–26 enable clinicians to generate an overall risk score based on a composite of traditional CVD risk factors. We assessed the performance of several of the most commonly used CVD risk functions, including the FHS (Framingham Heart Study) functions for hard coronary heart disease (FHS CHD) and atherosclerotic CVD (FHS ASCVD) and the American College of Cardiology/American Heart Association (ACC/AHA) pooled cohorts equations (ACC/AHA ASCVD). To assess the functions in a cohort of HIV-infected men, we compared regression coefficients and calculated calibration and discrimination of the established functions. We hypothesized that the established CVD risk functions would underestimate risk in HIV-infected individuals.
The data, analytic methods, and study materials will be made available to other researchers for purposes of reproducing the results 1 year after the dataset is locked through Dataverse (https://dataverse.harvard.edu/).
FHS and ACC/AHA Functions
Three established CVD risk functions were evaluated: the FHS hard CHD function (FHS CHD),20 the 2008 FHS global CVD function (FHS ASCVD),21 and the 2013 ACC/AHA function (ACC/AHA ASCVD).25 The FHS global CVD function was developed for a composite CVD outcome and was modified for the present study to ASCVD using the algorithm described in the Outcomes section and in the original 2008 manuscript.19 Risk factors included in the FHS functions are age, diabetes mellitus, smoking, blood pressure category, total cholesterol, and high-density lipoprotein (HDL). Risk factors included in the ACC/AHA function are age, diabetes mellitus, smoking, systolic blood pressure, total cholesterol, HDL, and antihypertensive treatment. We adapted the 10-year risk prediction functions to 5-year functions to align with the study follow-up period. Given that the proportional hazards assumption was met for variables in the Cox models, we developed a 5-year risk model for each function by (1) applying the Cox regression coefficients from the 10-year risk models to the 5-year risk models, and (2) replacing in the model 10-year baseline survival (event-free) estimates at the average values of the risk factors with 5-year baseline survival (event-free) estimates at the average values of the risk factors.19,20 The 5-year functions are included in the Appendix in the online-only Data Supplement.
The HIV population was identified from the Partners HIV cohort, an observational clinical care cohort of HIV-infected individuals receiving longitudinal HIV care through the Partners HealthCare System in Boston, Massachusetts. Data are derived from the Research Patient Data Registry, a data registry that includes comprehensive clinical information for all patients from the Partners HealthCare System. HIV-infected patients were initially identified from the Research Patient Data Registry by the presence of ≥1 International Classification of Diseases-Ninth Revision-Clinical Modification (ICD-9-CM) code for HIV (042 or V08). The HIV cohort was then limited to patients with ≥4 ICD-9-CM codes for HIV based on a validation study that demonstrated this definition of HIV status to be 88.9% sensitive and 98.6% specific for HIV infection based on physician medical record review. We limited the analyses to men because of low event numbers among women. Patients selected for inclusion were required to have ≥1 clinical encounter in calendar years 2006 to 2008, a blood pressure measurement available in 2006 to 2008, lipid laboratory values available in calendar years 2004 to 2008, smoking status available at baseline (2006–2008), and first HIV code that occurred before the start of observation for each individual. Exclusion criteria included history of a CVD event before the start of observation, <30 or >74 years of age for the FHS CHD and ASCVD functions, and <40 or >79 years of age for the ACC/AHA ASCVD function. The medical records of several prespecified subgroups in whom participants did not meet eligibility criteria by automated review were manually reviewed by a physician or research nurse to determine whether patients met inclusion criteria for the study; these included patients with exactly 3 ICD-9-CM codes for HIV, patients with the first HIV diagnosis after the observation start date, and patients missing 1 component of the CVD risk functions. The start of observation occurred on the date of risk score calculation, during a window period from January 1, 2006, to December 31, 2008. Participants who did not experience the outcome were censored at the earliest of noncardiovascular death, last follow-up visit, or 5 years after an individual’s start of observation. The study was approved by the Partners Human Research Committee, and no informed consent was required.
Calculation of Predicted CVD Risk
The baseline visit (visit of the risk score calculation) was the participant’s first visit that occurred between January 1, 2006, and December 31, 2008, that had nonmissing values for the risk factors necessary to generate a risk score. Age used was at baseline for each individual patient. Blood pressure data were obtained through electronic health record health maintenance–coded field data and by query of free text notes. Total and HDL cholesterol were obtained from laboratory data. Diabetes mellitus diagnosis was validated for this study and was defined as the presence of ≥3 ICD-9-CM codes of 250 and all subtypes (sensitivity, 86.2%; specificity, 93.0%). Smoking status was obtained through application of an natural language processing-based algorithm developed and validated in the Partners HIV cohort.27 For the ACC/AHA function, white coefficients were used because this race/ethnicity was the most common and because there was no evidence of an interaction between race and traditional CVD risk factors on the outcomes, as discussed later.
Hard CHD was defined as MI (ICD-9-CM code 410 and all subtypes) or coronary death. ASCVD was defined as MI (ICD-9-CM code 410 and all subtypes), stroke (ICD-9-CM codes 433–434 and all subtypes), or coronary death. The 2008 FHS function is for global CVD. Before applying it to the HIV sample, the function was adapted for the outcome of ASCVD for this analysis by using the adjustment factor of the ratio of the number of patients in the cohort with ASCVD to the number of patients with global CVD, following the methodology as outlined in the 2008 FHS global CVD publication.21 Global CVD is defined as MI (ICD-9-CM code 410 and all subtypes), stroke (ICD-9-CM codes 433–434 and all subtypes), coronary death, coronary insufficiency (ICD-9-CM codes 411, 412, and 414, and all subtypes), angina (inpatient ICD-9-CM code 411.1x), transient ischemic attack (ICD-9-CM code 435 and all subtypes), peripheral artery disease (ICD-9-CM codes 440.2x, 440.3x, and 440.4x), or heart failure (ICD-9-CM code 428 and all subtypes). Outcomes were identified by the presence of an ICD code for the relevant outcome that occurred after the start of observation; only the first code was included. ICD-based ascertainment of outcome events was validated by reviewing a randomly selected subset of patients with and without ICD codes for MI and stroke. Medical record review by a trained clinical nurse was the gold standard. For MI, sensitivity was 81.5%, and specificity was 86.3%. For stroke, sensitivity was 86.7%, and specificity was 78.8%. Coronary death data are obtained from the Partners Research Patient Data Registry, data on inpatient deaths are provided by the hospital, and data on out-of-hospital deaths are obtained from the Social Security Administration’s Death Master File. The Death Master File does not contain cause of death, so for the present study, medical records for all patients who died were individually reviewed by a physician to determine whether the cause of death was coronary.
Analyses described next were carried out separately for each of the 3 risk functions (FHS CHD, FHS ASCVD, and ACC/AHA ASCVD) and were limited to men because of low event numbers among women. Race/ethnicity groups were combined in the analyses because there were no significant interactions between race and the other risk factors on the outcomes at the 0.10 level of significance. We used white coefficients for the ACC/AHA function in the primary analysis. FHS functions were developed mostly on white participants, and there are no separate models for other races. In sensitivity analyses, we repeated all analyses (1) applying black coefficients for the ACC/AHA function to the entire HIV cohort, and (2) stratifying the analysis by race. For the race-stratified analyses, we categorized patients as black only, white only, or nonblack (white, Asian, Hispanic, other, or unknown) in separate analyses, using white coefficients for the white-only and nonblack groups and black coefficients for the black-only group.
Comparison of Coefficients
Cox proportional-hazards modeling was used to generate regression coefficients for each of the risk factors using data from the HIV cohort. Regression coefficients generated for the HIV cohort were descriptively compared to those previously generated for each function from FHS data. The coefficients generated from the HIV data were used to generate an HIV function that represents the best prediction function based on risk factor and outcome data from the HIV cohort, using the same traditional CVD risk factors as those included in the original functions and not including HIV-related risk factors.20 We did not directly compare relative risks between the ACC/AHA ASCVD and HIV functions because of the difficulty of interpretation of relative risks for the interaction terms included in the ACC/AHA function.
In supplemental analyses, we generated a new model based on risk factor and outcome data from the HIV cohort for men and women combined. For each function, we included all original risk factors and interaction terms with sex for each risk factor. We then used backward selection to remove nonsignificant interaction terms 1 at a time using a 0.10 level of significance (the interaction term with the largest P value >0.10 was removed in the backward approach, and we reran the regressions; the process was repeated until all remaining interaction terms had P<0.10). Original risk factors were retained in the model regardless of their significance. Analyses were conducted using SAS version 9.4.
To assess the ability of each prediction function to separate individuals who experience an event from individuals who do not, we calculated the c statistic. For each outcome, 2 c statistics were calculated. The first applied the established function (FHS CHD, ACC/AHA ASCVD, or FHS ASCVD) to the HIV data, and the second applied the HIV function, developed using risk factor and outcome data from the HIV cohort rather than using FHS or ACC/AHA cohort data, to the HIV data.
For each of the 3 risk functions and for the HIV function, calibration (goodness of fit of predicted risk to observed risk) was assessed using the Demler et al modification28 of the D’Agostino and Nam29 χ2 statistic. The HIV cohort was divided into deciles of predicted 5-year risk of CVD according to each risk function. For calculation of the χ2 statistic, we collapsed deciles with small numbers of categories until all resulting groups contained ≥5 events as required by Demler et al. This strategy was used to ensure a stable χ2 statistic. For each resulting group, the mean predicted risk and the observed Kaplan–Meier estimate of event rate were calculated and compared using the Demler/D’Agostino-Nam χ2 statistic; a nonsignificant χ2 (P value >0.05) indicates good calibration (ie, good fit of predicted risk to observed risk). In the event of poor calibration of the published risk models to the HIV data, recalibration was performed by replacing the mean values of risk factors and average incidence rates used from the established functions with those derived from the HIV cohort as discussed in D’Agostino et al.20 Calibration was then recalculated in a similar manner where participants were grouped in the following 3 categories of predicted risk instead of deciles: <5%, 5 to 7.5%, >7.5%.
Table 1 shows demographic and clinical characteristics of the HIV cohort among men 30 to 74 years of age for all individuals and according to CHD event status. The cohort was comprised of 1272 men, with median follow-up time of 4.4 years. Mean HDL was 43.7 mg/dL among patients without CHD events versus 40.7 mg/dL among those with events, and the proportion of patients with optimal or normal blood pressure was 60.9% among patients without CHD events versus 54.2% among those with events. More than 45% of patients were smokers; among patients with CHD events, 60.4% were smokers (versus 44.9% among those without events). Diabetes mellitus was present in 13.5%, with a rate of 27.1% among patients with CHD events (versus 13.0% among patients without events). Among all patients, 94.6% were receiving antiretroviral therapy. Median CD4 cell count was 457; among patients without CHD events, median CD4 count was 457 (versus 348 among those with events). The rate of viral suppression was 77.1% overall, 77.5% among patients without CHD events, and 66.7% among patients with CHD events.
|All (N=1272)||No CHD Event (N=1224)||CHD Event (N=48)|
|Age, y, mean±SD (N)||51.2±8.7 (1272)||51.1±8.7 (1224)||55.2±8.1 (48)|
|Age, y, min, max||30.2, 73.9||30.2, 73.9||34.9, 72.3|
|Race/ethnicity, % (n/N)|
|White||60.6 (771/1272)||60.5 (741/1224)||62.5 (30/48)|
|Black||19.7 (250/1272)||19.7 (241/1224)||18.8 (9/48)|
|Hispanic||13.8 (176/1272)||13.7 (168/1224)||16.7 (8/48)|
|Other/unknown||5.9 (75/1272)||6.0 (74/1224)||2.1 (1/48)|
|Total cholesterol, mg/dL, mean±SD||179.0±45.7||178.9±45.6||182.8±47.8|
|Total cholesterol, mg/dL, % (n/N)|
|<160||33.8 (430/1272)||33.5 (410/1224)||41.7 (20/48)|
|160–199||36.9 (470/1272)||37.4 (458/1224)||25.0 (12/48)|
|200–239||20.9 (266/1272)||21.0 (257/1224)||18.8 (9/48)|
|240–279||6.2 (79/1272)||6.0 (74/1224)||10.4 (5/48)|
|≥280||2.1 (27/1272)||2.0 (25/1224)||4.2 (2/48)|
|HDL cholesterol, mg/dL, mean±SD||43.6±14.2||43.7±14.3||40.7±11.3|
|HDL cholesterol, mg/dL, % (n/N)|
|<35||25.9 (329/1272)||25.6 (313/1224)||33.3 (16/48)|
|35–44||31.6 (402/1272)||31.7 (388/1224)||29.2 (14/48)|
|45–49||14.4 (183/1272)||14.1 (173/1224)||20.8 (10/48)|
|50–59||16.7 (212/1272)||16.8 (206/1224)||12.5 (6/48)|
|≥60||11.5 (146/1272)||11.8 (144/1224)||4.2 (2/48)|
|Systolic blood pressure, mean±SD||123.6±14.8||123.5±14.6||127.2±17.0|
|Treated systolic blood pressure, mean±SD (n)||127.5±16.7 (370)||127.5±16.6 (346)||127.5±19.3 (24)|
|Untreated systolic blood pressure, mean±SD (n)||122.1±13.6 (902)||121.9±13.5 (878)||127.0±14.8 (24)|
|Blood pressure categories, % (n/N)|
|Optimal (SBP <120, DBP <80)||31.0 (394/1272)||31.2 (382/1224)||25.0 (12/48)|
|Normal (SBP <130, DBP <85)||29.6 (377/1272)||29.7 (363/1224)||29.2 (14/48)|
|High normal (SBP <140, DBP <90)||17.0 (216/1272)||16.8 (206/1224)||20.8 (10/48)|
|Stage I HTN (SBP <160, DBP <100)||17.4 (221/1272)||17.4 (213/1224)||16.7 (8/48)|
|Stage II–IV HTN (SBP ≥160, DBP ≥100)||5.0 (64/1272)||4.9 (60/1224)||8.3 (4/48)|
|Antihypertensive medication, % (n/N)||29.1 (370/1272)||28.3 (346/1224)||50.0 (24/48)|
|Smoking, % (n/N)||45.4 (578/1272)||44.9 (549/1224)||60.4 (29/48)|
|Diabetes mellitus, % (n/N)||13.5 (172/1272)||13.0 (159/1224)||27.1 (13/48)|
|CD4 count, median (Q1, Q3)||457 (277, 672)||457 (281, 672)||348 (222, 687)|
|CD4 count <200 cells/mm3, % (n/N)||14.9 (176/1183)||14.7 (168/1140)||18.6 (8/43)|
|HIV viral load <400 copies/mL, % (n/N)||77.1 (804/1043)||77.5 (778/1004)||66.7 (26/39)|
|Log HIV viral load, median (Q1, Q3)||4.2 (3.5, 4.7)||4.2 (3.5, 4.7)||4.5 (3.9, 4.9)|
|ART use, % (n/N)||94.6 (1203/1272)||94.5 (1157/1224)||95.8 (46/48)|
|Duration of follow-up, y,– median (Q1, Q3)||4.4 (3.0, 5.0)||4.5 (3.1, 5.0)||1.7 (0.6, 3.8)|
|5-Year hard CHD rate, % (n)||3.8 (48)||—||—|
|5-Year hard CHD incidence rate per 1000 PY||10.0||—||—|
|5-Year ASCVD rate, % (n)||6.1 (78)||—||—|
|5-Year ASCVD incidence rate per 1000 PY||16.4||—||—|
The 5-year hard CHD event rate was 3.8% (48/1272), and the 5-year ASCVD event rate was 6.1% (78/1272). Incidence rates for hard CHD and ASCVD are shown in Table 1. Median risk scores for each function are shown in Table I in the online-only Data Supplement and include predicted risk scores based on established functions, observed risk, and predicted risk scores based on the Cox models developed from the HIV cohort (HIV function).
Table II in the online-only Data Supplement shows demographic and clinical characteristics of the HIV cohort among women 30 to 74 years of age overall and according to CHD event status.
Comparison of Coefficients
Table 2 shows coefficients and 2-sided 95% confidence intervals for risk factors from the HIV cohort’s regression model and also displays published coefficients derived from the original cohort’s (FHS or ACC/AHA) model for each function among men. Qualitative comparison of coefficients was performed, taking into consideration that interpretation of results is limited by a low number of events. The majority of the coefficients developed from the HIV function did not achieve statistical significance (confidence interval of the coefficient contained 0) for each of the 3 established functions evaluated. For hard CHD, increasing total cholesterol category was associated with higher hard CHD risk in both the FHS and HIV-developed models. Being in the lowest HDL category (<35) yielded the largest hard CHD risk, and being in the highest HDL category (≥60) was associated with the lowest risk in both hard CHD models. Hypertension was associated with a higher risk of hard CHD in both models. As anticipated, age, diabetes mellitus, and smoking were associated with higher hard CHD risk in both models.
|Risk Factor||Original Function||HIV Function (95% CI)|
|Age||0.05||0.048 (0.013, 0.084)|
|Optimal, SBP <120, DBP <80||0.09||−0.209 (−0.986, 0.568)|
|Normal, SBP <130, DBP <85||Reference||Reference|
|High normal, SBP <140, DBP <90||0.42||−0.027 (−0.85, 0.797)|
|Stage I HTN, SBP <160, DBP <100||0.66||−0.271 (−1.154, 0.612)|
|Stage II–IV HTN, SBP ≥160, DBP ≥100)||0.9||0.406 (−0.719, 1.532)|
|<160||−0.38||0.603 (−0.13, 1.335)|
|200–239||0.57||0.278 (−0.598, 1.153)|
|240–279||0.74||0.901 (−0.149, 1.952)|
|≥280||0.83||1.427 (−0.095, 2.95)|
|<35||0.61||0.045 (−0.781, 0.871)|
|35–44||0.37||−0.435 (−1.255, 0.385)|
|50–59||0||−0.726 (−1.748, 0.297)|
|≥60||−0.46||−1.345 (−2.875, 0.185)|
|Diabetes mellitus||0.53||0.71 (0.05, 1.369)|
|Smoking||0.73||0.804 (0.213, 1.395)|
|Ln age||12.344||1.855 (−34.058, 37.768)|
|Ln total cholesterol||11.853||5.641 (−23.64, 34.921)|
|Ln HDL cholesterol||−7.990||−9.173 (−29.337, 10.991)|
|Ln treated SBP||1.797||0.463 (−1.357, 2.283)|
|Ln untreated SBP||1.764||0.417 (−1.418, 2.252)|
|Smoking||7.837||1.384 (−11.186, 13.954)|
|Diabetes mellitus||0.658||0.406 (−0.128, 0.939)|
|Ln age × Ln total cholesterol||−2.664||−1.303 (−8.6, 5.994)|
|Ln age × Ln HDL cholesterol||1.769||2.102 (−2.921, 7.125)|
|Ln age × smoking||−1.795||−0.236 (−3.359, 2.887)|
|Ln age × Ln age||NA||NA|
|Ln age||3.061||2.886 (1.352−4.419)|
|Ln total cholesterol||1.124||0.471 (−0.491, 1.434)|
|Ln HDL cholesterol||−0.933||−0.761 (−1.471, −0.051)|
|Ln treated SBP||1.999||0.775 (−1.057, 2.606)|
|Ln untreated SBP||1.933||0.713 (−1.133, 2.559)|
|Diabetes mellitus||0.574||0.38 (−0.161, 0.921)|
For the ACC/AHA ASCVD function, age, total cholesterol, systolic blood pressure, smoking, and diabetes mellitus were associated with higher ASCVD risk in both the ACC/AHA and HIV models. For both models, as expected, increased HDL was associated with lower ASCVD risk.
For the FHS ASCVD function, all risk factors had similar relative effects in the FHS and HIV models, with increased age, total cholesterol, systolic blood pressure, smoking, and diabetes mellitus associated with higher risk and HDL associated with lower risk.
Table 3 shows the c statistic, a measurement of discrimination, and its 2-sided 95% confidence interval for each function applied to the HIV cohort. In the table, the Original Function rows indicate the c statistic derived from applying the FHS CHD, ACC/AHA ASCVD, or FHS ASCVD function to the HIV cohort’s data, respectively. The HIV Function row indicates the c statistic derived from a Cox regression model developed from the HIV cohort using the same variables as contained in the relevant FHS or ACC/AHA function but using risk factor values and outcome data measured on the HIV cohort (HIV function). Discrimination of all published functions when applied to the HIV cohort was suboptimal (Original function), with c statistics of 0.68 for FHS CHD, 0.65 for ACC/AHA ASCVD, and 0.67 for FHS ASCVD. Application of the Cox regression model developed from the HIV cohort to the HIV cohort (HIV function) resulted in improved but not optimized discrimination for most groups (c statistics of 0.73 for FHS CHD, 0.66 for ACC/AHA ASCVD, and 0.67 for FHS ASCVD).
|FHS CHD||ACC/AHA ASCVD||FHS ASCVD|
|Original function||0.68 (0.61–0.75)||0.65 (0.59–0.71)||0.67 (0.61–0.73)|
|HIV function||0.73 (0.67–0.81)||0.66 (0.60–0.73)||0.67 (0.61–0.73)|
Calibration and Observed Versus Predicted Risk
Figure 1 shows observed and predicted 5-year risk for each of the functions by decile of predicted risk for all participants. For all functions, observed risk exceeded predicted risk for most deciles of predicted risk. Figure 2 shows observed and predicted 5-year risk for each of the functions by predicted risk group (<5%, 5%–7.5%, and >7.5%). Observed risk exceeded predicted risk for all categories in all functions except for >7.5% predicted risk for the FHS hard CHD function.
Table I in the online-only Data Supplement shows median risk scores by function and compares predicted versus observed risk. For all 3 functions, observed risk exceeded predicted risk, whether predicted risk was calculated using the traditional functions or the HIV function. All predicted risk scores, however, were greater and closer to the observed risk when the HIV function was applied versus when the traditional functions were applied.
The calibration, or goodness of fit, of the FHS and ACC/AHA functions applied to the HIV cohort was assessed using the calibration χ2 statistic. For the 3 functions assessed, calibration was poor and reflects inadequate fit of the general population functions to the HIV cohort. For the FHS CHD function, the calibration χ2 statistic was 13.6 (P=0.019, 5 degrees of freedom). For the ACC/AHA function, the calibration χ2 statistic was 23.9 (P=0.001, 7 degrees of freedom). For the FHS ASCVD function, the calibration χ2 statistic was 24.6 (P=0.0004, 6 degrees of freedom).
We recalibrated the FHS or ACC/AHA model to attempt to improve the model fit of these functions by using baseline survival and mean risk factor values from the HIV cohort instead of the FHS or ACC/AHA cohorts’ values. After recalibration, goodness of fit remained poor for all functions, and model performance did not improve (data not shown).
In the primary analysis for the ACC/AHA ASCVD function, white coefficients were used given that there was no evidence of any interaction between race and traditional CVD risk factors in the HIV cohort. We repeated all analyses using black coefficients for the ACC/AHA function and generated similar results. To further confirm that each function poorly discriminated and underestimated risk in the HIV cohort, we conducted analyses stratified by race and showed that discrimination remained moderate and calibration remained poor (Tables III and IV in the online-only Data Supplement).
In supplemental analyses, we generated a new model (HIV function) among men and women combined, including significant interaction terms with sex for each risk factor. Results of this analysis are shown in Table V in the online-only Data Supplement.
We evaluated 3 established cardiovascular risk prediction functions in an HIV cohort and found that the functions systematically underestimate CVD risk in HIV. Discrimination and calibration were both suboptimal when applying the functions to a cohort of largely antiretroviral therapy–treated men engaged in HIV care, indicating that risk prediction algorithms developed for use in the general population may underperform and may be inadequate in their ability to appropriately order CVD risk and accurately calculate predicted risk. Our findings suggest that established CVD risk functions do not provide an accurate estimation of risk in the setting of HIV disease and may fail to identify patients at elevated CVD risk who would benefit from aggressive risk reduction. Adaptation of current CVD risk functions with the incorporation of new risk factors is likely to be needed for safe and accurate risk estimation in HIV.
The question of transportability of CVD risk prediction functions to populations that differ from those in which they are developed has received considerable attention.20 To date, the FHS functions have been the most widely used to quantify the absolute risk of developing a specific CVD outcome for an individual patient over a given time period.20,30 The release of ACC/AHA guidelines on CVD risk assessment and cholesterol treatment represents a turning point in the approach to CVD prevention in the general population,25,26,31 with the development of a new CVD risk prediction function derived from a more representative population and utilizing the broader outcome of ASCVD. The applicability of established functions in the setting of HIV, however, remains incompletely understood. The ACC/AHA guidelines specifically call for validation in diverse settings,25 and in this analysis, we find established prediction functions to be inadequate when applied to individuals with HIV.
Developing accurate CVD risk functions in HIV-infected individuals has been identified as a priority by both HIV and cardiovascular advocacy groups.25,26,32 The hypothesis that conventional CVD risk prediction functions may not perform accurately in HIV is based on proposed mechanistic factors. Although traditional CVD risk factors are prevalent in HIV and contribute to CVD risk,6 extensive data now suggest that HIV-specific inflammation and immune activation play prominent roles in mediating CVD risk.11,13,14 Even in virally suppressed patients, inflammation and immune activation continue to be abnormal,33 and persistent chronic inflammation may increase long-term CVD risk and severity.34,35 Adding to the complexity of the mechanism of HIV-associated CVD, several individual HIV medications have been shown to be associated with increased CVD risk,16,17 although whether they mediate risk via effects on traditional risk factors versus novel mechanisms remains unclear and is likely to differ by drug. Established CVD risk algorithms were modeled after risk factors common in the general population and do not reflect these unique mechanistic factors thought to drive HIV-related CVD.
Prior studies in HIV have evaluated the correlation between CVD risk scores and subclinical atherosclerosis36 and suggest that FHS functions underestimate the risk of acute MI37 and stroke.38 Data from the HOPS (HIV Outpatient Study)39 demonstrated accurate risk estimation but suboptimal discrimination for the FHS global CVD function, underestimation and reasonable discrimination by the ACC/AHA and D:A:D (Data collection on Adverse Events of Anti-HIV Drugs) functions,40,41 and overestimation with poor discrimination for the SCORE (Systematic Coronary Risk Evaluation) function.42 A recent study from the Centers for AIDS Research Network of Integrated Clinical Systems cohort found the ACC/AHA function to adequately discriminate MI risk and to be moderately calibrated, with variability in performance by demographic subgroups.43 Additionally, incorporation of HIV-specific variables did not improve the risk score.43 Our study builds on prior knowledge by providing a formal assessment of several established functions’ performance in terms of discrimination, calibration, and comparison of regression coefficients and provides new knowledge by evaluating the ACC/AHA and 2 FHS functions in a cohort of HIV-infected individuals with the ability to evaluate the outcome of ASCVD, including MI and stroke.
Our data support the hypothesis that established CVD risk functions underestimate risk for HIV-infected men. Our assessment of the models’ discriminatory capacity indicated that all the functions suboptimally ordered risk in the HIV cohort; the c statistics, which represent the probability that patients who develop the outcome will be assigned a higher risk than those who do not, were lower than the standard threshold for all 3 functions when applied to the HIV cohort. Observed risk was greater than predicted risk across all 3 risk functions and for most deciles and groups of predicted risk. The degree of agreement between predicted and observed risk was formally studied through assessment of calibration, with large χ2 statistics and significant P values indicating poor model fit for all 3 functions. Comparison of regression coefficients, applying the original function versus the HIV function, indicates different magnitudes and, at times, directions of the effect of traditional risk factors on cardiovascular outcomes, although qualitative comparisons are limited by a relatively low number of events.
To attempt to improve model performance, we generated an HIV function, which represents the best function that can be developed based on risk factor and outcome data from the HIV cohort but using risk factors identical to those included in the established function. We evaluated these best HIV functions—developed from the 3 established functions—in terms of traditional risk factors, discrimination, and calibration. For these newly developed functions, we found that few of the regression coefficients for traditional CVD risk factors achieved significance. Discrimination, as indicated by the c statistic, improved for the HIV functions but remained inadequate overall, and recalibration only modestly improved model fit for each HIV function. These findings indicate that the best model we can develop, directly computed from the HIV data, only marginally improves risk prediction, and that additional research is needed—likely including the addition of new variables—to develop an adequate risk model in the HIV population.
Together these data suggest that cardiovascular risk prediction using established algorithms developed for the general population remains suboptimal in the setting of HIV disease, and that as a first step, the discriminatory capacity of the models needs to be improved. Our findings of the models’ moderate to poor discriminatory capacity are consistent with the possibility of an unaccounted-for risk factor that is relevant to HIV-infected individuals but not to the populations from which the functions were developed. This missing variable is likely to be related to HIV-associated inflammation and immune activation, which have been shown to directly impact cardiovascular risk in this population.44–46 Although HIV-related, novel CVD risk factors have been demonstrated to be significantly associated with CVD outcomes, it is unknown whether the incorporation of such factors into established risk prediction functions will improve the performance of the functions. Moreover, risk factors considered for inclusion should be clinically relevant to facilitate widespread implementation of a new HIV-specific function. Although the primary purpose of this study was to investigate the role of previously studied traditional CVD risk factors in risk prediction in HIV, future work is planned to evaluate the incorporation of HIV-specific variables, including HIV disease parameters and medications as well as HIV itself, into tailored risk prediction algorithms.
Our study has several limitations, including the possibility of inaccurate outcome ascertainment because of missing data or incorrect classification. CVD outcomes were validated for the current study; as 10-year outcome data become available, adjudication of events is planned. The follow-up time of 5 years (versus 10 years) was selected to balance and optimize completeness and accuracy of electronic health record data (which has improved over time in the Research Patient Data Registry) with longer duration of follow-up. Although the lower number of events limits interpretation of current results, we are prospectively collecting 10-year data and anticipate an increasing number of events as more follow-up time is accumulated. Additionally, we did not include women in the analysis because of the difficulty of interpreting results because of low event numbers. Thus, findings are generalizable to HIV-infected men. Future analyses of 10-year events including women are planned.
In conclusion, cardiovascular risk prediction using established general population algorithms was found to be inaccurate in HIV infection, with consistent underestimation of risk. In this study using a longitudinal HIV clinical care cohort, we assessed the performance of 2 established FHS functions and the ACC/AHA function and found that all functions resulted in poor model fit when applied to the HIV population. Our findings suggest that current application of established CVD risk functions in clinical practice may not result in accurate CVD risk assessment, and that reliance on such measures may fail to identify high-risk HIV-infected individuals who would benefit from aggressive cardiovascular risk modification. Refining current CVD risk functions through the addition of novel HIV-related factors, including HIV, may improve CVD risk function performance. The lack of transportability of established CVD functions to HIV underscores the need for tailored CVD risk prediction strategies for this at-risk population to manage and prevent one of the most serious complications of long-term HIV infection.
The authors are grateful to the Partners HealthCare Research Patient Data Registry group for facilitating use of their database and to Steven McDermott, RN, for assistance with validation studies.
Sources of Funding
This work was supported by the National Institutes of Health (R01HL132786 to V.A.T. and R.B.D., R56HL125029 to V.A.T. and R.B.D., P30DK040561 to S.K.G., and K24DK080140 to J.B.M.).
Dr Grinspoon has received research support through his institution from KOWA Pharmaceuticals. No conflict is identified.
Currier JS. Cardiovascular risk associated with HIV therapy.J Acquir Immune Defic Syndr. 2002; 31(suppl 1):S16–S23.CrossrefMedlineGoogle Scholar
Triant VA, Lee H, Hadigan C, Grinspoon SK. Increased acute myocardial infarction rates and cardiovascular risk factors among patients with human immunodeficiency virus disease.J Clin Endocrinol Metab. 2007; 92:2506–2512. doi: 10.1210/jc.2006-2190.CrossrefMedlineGoogle Scholar
Freiberg MS, Chang CC, Kuller LH, Skanderson M, Lowy E, Kraemer KL, Butt AA, Bidwell Goetz M, Leaf D, Oursler KA, Rimland D, Rodriguez Barradas M, Brown S, Gibert C, McGinnis K, Crothers K, Sico J, Crane H, Warner A, Gottlieb S, Gottdiener J, Tracy RP, Budoff M, Watson C, Armah KA, Doebler D, Bryant K, Justice AC. HIV infection and the risk of acute myocardial infarction.JAMA Intern Med. 2013; 173:614–622. doi: 10.1001/jamainternmed.2013.3728.CrossrefMedlineGoogle Scholar
Brown TT, Cole SR, Li X, Kingsley LA, Palella FJ, Riddler SA, Visscher BR, Margolick JB, Dobs AS. Antiretroviral therapy and the prevalence and incidence of diabetes mellitus in the multicenter AIDS cohort study.Arch Intern Med. 2005; 165:1179–1184. doi: 10.1001/archinte.165.10.1179.CrossrefMedlineGoogle Scholar
Friis-Møller N, Weber R, Reiss P, Thiébaut R, Kirk O, d’Arminio Monforte A, Pradier C, Morfeldt L, Mateu S, Law M, El-Sadr W, De Wit S, Sabin CA, Phillips AN, Lundgren JD; DAD study group. Cardiovascular disease risk factors in HIV patients–association with antiretroviral therapy: results from the DAD study.AIDS. 2003; 17:1179–1193. doi: 10.1097/01.aids.0000060358.78202.c1.CrossrefMedlineGoogle Scholar
Grinspoon S, Carr A. Cardiovascular risk and body-fat abnormalities in HIV-infected adults.N Engl J Med. 2005; 352:48–62. doi: 10.1056/NEJMra041811.CrossrefMedlineGoogle Scholar
Hadigan C, Meigs JB, Corcoran C, Rietschel P, Piecuch S, Basgoz N, Davis B, Sax P, Stanley T, Wilson PW, D’Agostino RB, Grinspoon S. Metabolic abnormalities and cardiovascular disease risk factors in adults with human immunodeficiency virus infection and lipodystrophy.Clin Infect Dis. 2001; 32:130–139. doi: 10.1086/317541.CrossrefMedlineGoogle Scholar
Riddler SA, Smit E, Cole SR, Li R, Chmiel JS, Dobs A, Palella F, Visscher B, Evans R, Kingsley LA. Impact of HIV infection and HAART on serum lipids in men.JAMA. 2003; 289:2978–2982. doi: 10.1001/jama.289.22.2978.CrossrefMedlineGoogle Scholar
Seaberg EC, Muñoz A, Lu M, Detels R, Margolick JB, Riddler SA, Williams CM, Phair JP; Multicenter AIDS Cohort Study. Association between highly active antiretroviral therapy and hypertension in a large cohort of men followed from 1984 to 2003.AIDS. 2005; 19:953–960.CrossrefMedlineGoogle Scholar
Helleberg M, Afzal S, Kronborg G, Larsen CS, Pedersen G, Pedersen C, Gerstoft J, Nordestgaard BG, Obel N. Mortality attributable to smoking among HIV-1-infected individuals: a nationwide, population-based cohort study.Clin Infect Dis. 2013; 56:727–734. doi: 10.1093/cid/cis933.CrossrefMedlineGoogle Scholar
El-Sadr WM, Lundgren J, Neaton JD, Gordin F, Abrams D, Arduino RC, Babiker A, Burman W, Clumeck N, Cohen CJ, Cohn D, Cooper D, Darbyshire J, Emery S, Fatkenheuer G, Gazzard B, Grund B, Hoy J, Klingman K, Losso M, Markowitz N, Neuhaus J, Phillips A, Rappoport C; Strategies for Management of Antiretroviral Therapy Study Group. CD4+ count-guided interruption of antiretroviral treatment.N Engl J Med. 2006; 355:2283–2296.CrossrefMedlineGoogle Scholar
Kaplan RC, Sinclair E, Landay AL, Lurain N, Sharrett AR, Gange SJ, Xue X, Hunt P, Karim R, Kern DM, Hodis HN, Deeks SG. T cell activation and senescence predict subclinical carotid artery disease in HIV-infected women.J Infect Dis. 2011; 203:452–463. doi: 10.1093/infdis/jiq071.CrossrefMedlineGoogle Scholar
Kuller LH, Tracy R, Belloso W, De Wit S, Drummond F, Lane HC, Ledergerber B, Lundgren J, Neuhaus J, Nixon D, Paton NI, Neaton JD; INSIGHT SMART Study Group. Inflammatory and coagulation biomarkers and mortality in patients with HIV infection.PLoS Med. 2008; 5:e203. doi: 10.1371/journal.pmed.0050203.CrossrefMedlineGoogle Scholar
Phillips AN, Carr A, Neuhaus J, Visnegarwala F, Prineas R, Burman WJ, Williams I, Drummond F, Duprez D, Belloso WH, Goebel FD, Grund B, Hatzakis A, Vera J, Lundgren JD. Interruption of antiretroviral therapy and risk of cardiovascular disease in persons with HIV-1 infection: exploratory analyses from the SMART trial.Antivir Ther. 2008; 13:177–187.CrossrefMedlineGoogle Scholar
Triant VA, Grinspoon SK. Immune dysregulation and vascular risk in HIV-infected patients: implications for clinical care.J Infect Dis. 2011; 203:439–441. doi: 10.1093/infdis/jiq084.CrossrefMedlineGoogle Scholar
Friis-Moller N, Reiss P, Sabin CA, Weber R, Monforte A, El-Sadr W, Thiebaut R, De Wit S, Kirk O, Fontas E, Law MG, Phillips A, Lundgren JD. Class of antiretroviral drugs and the risk of myocardial infarction.N Engl J Med. 2007; 356:1723–1735.CrossrefMedlineGoogle Scholar
Sabin CA, Worm SW, Weber R, Reiss P, El-Sadr W, Dabis F, De Wit S, Law M, D’Arminio Monforte A, Friis-Moller N, Kirk O, Pradier C, Weller I, Phillips AN, Lundgren JD. Use of nucleoside reverse transcriptase inhibitors and risk of myocardial infarction in HIV-infected patients enrolled in the D:A:D study: a multi-cohort collaboration.Lancet. 2008; 371:1417–1426.CrossrefMedlineGoogle Scholar
D’Agostino RBCardiovascular risk estimation in 2012: lessons learned and applicability to the HIV population.J Infect Dis. 2012; 205(suppl 3):S362–S367. doi: 10.1093/infdis/jis196.CrossrefMedlineGoogle Scholar
Anderson KM, Wilson PW, Odell PM, Kannel WB. An updated coronary risk profile: a statement for health professionals.Circulation. 1991; 83:356–362.LinkGoogle Scholar
D’Agostino RB, Grundy S, Sullivan LM, Wilson P; CHDRP Group. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.JAMA. 2001; 286:180–187.CrossrefMedlineGoogle Scholar
D’Agostino RB, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB. General cardiovascular risk profile for use in primary care: the Framingham Heart Study.Circulation. 2008; 117:743–753. doi: 10.1161/CIRCULATIONAHA.107.699579.LinkGoogle Scholar
- 22. Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults. Executive summary of the third report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III).JAMA. 2001; 285:2486–2497.CrossrefMedlineGoogle Scholar
Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories.Circulation. 1998; 97:1837–1847.LinkGoogle Scholar
Wolf PA, D’Agostino RB, Belanger AJ, Kannel WB. Probability of stroke: a risk profile from the Framingham Study.Stroke. 1991; 22:312–318.LinkGoogle Scholar
Goff DC, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, Greenland P, Lackland DT, Levy D, O’Donnell CJ, Robinson JG, Schwartz JS, Shero ST, Smith SC, Sorlie P, Stone NJ, Wilson PW, Jordan HS, Nevo L, Wnek J, Anderson JL, Halperin JL, Albert NM, Bozkurt B, Brindis RG, Curtis LH, DeMets D, Hochman JS, Kovacs RJ, Ohman EM, Pressler SJ, Sellke FW, Shen WK, Smith SC, Tomaselli GF; American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.Circulation. 2014; 129(25 suppl 2):S49–S73. doi: 10.1161/01.cir.0000437741.48606.98.LinkGoogle Scholar
Stone NJ, Robinson JG, Lichtenstein AH, Bairey Merz CN, Blum CB, Eckel RH, Goldberg AC, Gordon D, Levy D, Lloyd-Jones DM, McBride P, Schwartz JS, Shero ST, Smith SC, Watson K, Wilson PW, Eddleman KM, Jarrett NM, LaBresh K, Nevo L, Wnek J, Anderson JL, Halperin JL, Albert NM, Bozkurt B, Brindis RG, Curtis LH, DeMets D, Hochman JS, Kovacs RJ, Ohman EM, Pressler SJ, Sellke FW, Shen WK, Smith SC, Tomaselli GF; American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.Circulation. 2014; 129(25 suppl 2):S1–S45. doi: 10.1161/01.cir.0000437738.63853.7a.LinkGoogle Scholar
Regan S, Meigs JB, Grinspoon SK, Triant VA. Determinants of smoking and quitting in HIV-infected individuals.PLoS One. 2016; 11:e0153103. doi: 10.1371/journal.pone.0153103.CrossrefMedlineGoogle Scholar
Demler OV, Paynter N, Cook NR. Tests of calibration and goodness-of-fit in survival settings.Statistics in Medicine. 2015; 34:1659–1680.CrossrefMedlineGoogle Scholar
D’Agostino RB, Nam BH. Evaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures. New York: Elsevier Science B.V.; 2004.Google Scholar
D’Agostino RB, Pencina MJ, Massaro JM, Coady S. Cardiovascular disease risk assessment: insights from Framingham.Glob Heart. 2013; 8:11–23. doi: 10.1016/j.gheart.2013.01.001.CrossrefMedlineGoogle Scholar
Lloyd-Jones DM, Morris PB, Ballantyne CM, Birtcher KK, Daly DD, DePalma SM, Minissian MB, Orringer CE, Smith SC; Writing Committee. 2016 ACC expert consensus decision pathway on the role of non-statin therapies for LDL-cholesterol lowering in the management of atherosclerotic cardiovascular disease risk: a report of the American College of Cardiology Task Force on Clinical Expert Consensus Documents.J Am Coll Cardiol. 2016; 68:92–125. doi: 10.1016/j.jacc.2016.03.519.CrossrefMedlineGoogle Scholar
Aberg JA, Gallant JE, Ghanem KG, Emmanuel P, Zingman BS, Horberg MA; Infectious Diseases Society of America. Primary care guidelines for the management of persons infected with HIV: 2013 update by the HIV medicine association of the Infectious Diseases Society of America.Clin Infect Dis. 2014; 58:e1–e34. doi: 10.1093/cid/cit665.CrossrefMedlineGoogle Scholar
French MA, King MS, Tschampa JM, da Silva BA, Landay AL. Serum immune activation markers are persistently increased in patients with HIV infection after 6 years of antiretroviral therapy despite suppression of viral replication and reconstitution of CD4+ T cells.J Infect Dis. 2009; 200:1212–1215. doi: 10.1086/605890.CrossrefMedlineGoogle Scholar
Hsue PY, Hunt PW, Schnell A, Kalapus SC, Hoh R, Ganz P, Martin JN, Deeks SG. Role of viral replication, antiretroviral therapy, and immunodeficiency in HIV-associated atherosclerosis.AIDS. 2009; 23:1059–1067. doi: 10.1097/QAD.0b013e32832b514b.CrossrefMedlineGoogle Scholar
Nordell AD, McKenna M, Borges ÁH, Duprez D, Neuhaus J, Neaton JD; INSIGHT SMART, ESPRIT Study Groups; SILCAAT Scientific Committee. Severity of cardiovascular disease outcomes among patients with HIV is related to markers of inflammation and coagulation.J Am Heart Assoc. 2014; 3:e000844. doi: 10.1161/JAHA.114.000844.LinkGoogle Scholar
Parra S, Coll B, Aragonés G, Marsillach J, Beltrán R, Rull A, Joven J, Alonso-Villaverde C, Camps J. Nonconcordance between subclinical atherosclerosis and the calculated Framingham risk score in HIV-infected patients: relationships with serum markers of oxidation and inflammation.HIV Med. 2010; 11:225–231. doi: 10.1111/j.1468-1293.2009.00766.x.CrossrefMedlineGoogle Scholar
Law MG, Friis-Møller N, El-Sadr WM, Weber R, Reiss P, D’Arminio Monforte A, Thiébaut R, Morfeldt L, De Wit S, Pradier C, Calvo G, Kirk O, Sabin CA, Phillips AN, Lundgren JD; D:A:D Study Group. The use of the Framingham equation to predict myocardial infarctions in HIV-infected patients: comparison with observed events in the D:A:D study.HIV Med. 2006; 7:218–230. doi: 10.1111/j.1468-1293.2006.00362.x.CrossrefMedlineGoogle Scholar
Mateen FJ, Post WS, Sacktor N, Abraham AG, Becker JT, Smith BR, Detels R, Martin E, Phair JP, Shinohara RT; Multicenter AIDS Cohort Study (MACS) Investigators. Long-term predictive value of the Framingham Risk Score for Stroke in HIV-positive vs HIV-negative men.Neurology. 2013; 81:2094–2102. doi: 10.1212/01.wnl.0000437296.97946.73.CrossrefMedlineGoogle Scholar
Thompson-Paul AM, Lichtenstein KA, Armon C, Palella FJ, Skarbinski J, Chmiel JS, Hart R, Wei SC, Loustalot F, Brooks JT, Buchacz K. Cardiovascular disease risk prediction in the HIV Outpatient Study.Clin Infect Dis. 2016; 63:1508–1516. doi: 10.1093/cid/ciw615.CrossrefMedlineGoogle Scholar
Friis-Møller N, Ryom L, Smith C, Weber R, Reiss P, Dabis F, De Wit S, Monforte AD, Kirk O, Fontas E, Sabin C, Phillips A, Lundgren J, Law M; D:A:D Study Group. An updated prediction model of the global risk of cardiovascular disease in HIV-positive persons: the Data-collection on Adverse Effects of Anti-HIV Drugs (D:A:D) study.Eur J Prev Cardiol. 2016; 23:214–223. doi: 10.1177/2047487315579291.CrossrefMedlineGoogle Scholar
Friis-Møller N, Thiébaut R, Reiss P, Weber R, Monforte AD, De Wit S, El-Sadr W, Fontas E, Worm S, Kirk O, Phillips A, Sabin CA, Lundgren JD, Law MG; D:A:D Study Group. Predicting the risk of cardiovascular disease in HIV-infected patients: the data collection on adverse effects of anti-HIV drugs study.Eur J Cardiovasc Prev Rehabil. 2010; 17:491–501. doi: 10.1097/HJR.0b013e328336a150.CrossrefMedlineGoogle Scholar
Conroy RM, Pyörälä K, Fitzgerald AP, Sans S, Menotti A, De Backer G, De Bacquer D, Ducimetière P, Jousilahti P, Keil U, Njølstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM; SCORE Project Group. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project.Eur Heart J. 2003; 24:987–1003.CrossrefMedlineGoogle Scholar
Feinstein MJ, Nance RM, Drozd DR, Ning H, Delaney JA, Heckbert SR, Budoff MJ, Mathews WC, Kitahata MM, Saag MS, Eron JJ, Moore RD, Achenbach CJ, Lloyd-Jones DM, Crane HM. Assessing and refining myocardial infarction risk estimation among patients with human immunodeficiency virus: a study by the Centers for AIDS Research Network of integrated clinical systems.JAMA Cardiol. 2017; 2:155–162. doi: 10.1001/jamacardio.2016.4494.CrossrefMedlineGoogle Scholar
Stein JH, Hsue PY. Inflammation, immune activation, and CVD risk in individuals with HIV infection.JAMA. 2012; 308:405–406. doi: 10.1001/jama.2012.8488.CrossrefMedlineGoogle Scholar
Plaeger SF, Collins BS, Musib R, Deeks SG, Read S, Embry A. Immune activation in the pathogenesis of treated chronic HIV disease: a workshop summary.AIDS Res Hum Retroviruses. 2012; 28:469–477. doi: 10.1089/aid.2011.0213.CrossrefMedlineGoogle Scholar
Nou E, Lo J, Grinspoon SK. Inflammation, immune activation, and cardiovascular disease in HIV.AIDS. 2016; 30:1495–1509. doi: 10.1097/QAD.0000000000001109.CrossrefMedlineGoogle Scholar