Cardiovascular Risk Prediction Functions Underestimate Risk in HIV Infection

Introduction

Editorial, see p 2215

Cardiovascular disease (CVD) risk prediction functions are widely used to predict CVD risk and prevent disease through identification of the highest-risk individuals who merit intensive risk factor modification. HIV-infected individuals face an increased CVD risk, with multiple studies demonstrating a 1.5- to 2-fold increased risk of myocardial infarction (MI) or stroke compared with non-HIV–infected individuals.^1–3 Although traditional CVD risk factors such as smoking, dyslipidemia, diabetes mellitus, and hypertension are prevalent in HIV,^2,4–10 emerging data support a prominent role of novel CVD risk factors related to chronic inflammation and immune activation in HIV-related CVD risk.^11–15 Moreover, specific HIV medications have been shown to alter CVD risk.^16,17 Established CVD risk functions were developed using risk factors common in the general population and do not reflect these unique mechanistic factors thought to drive HIV-related CVD. Consequently, existing functions may not accurately estimate risk in the HIV setting.¹⁸

The most widely applied CVD risk functions^19–26 enable clinicians to generate an overall risk score based on a composite of traditional CVD risk factors. We assessed the performance of several of the most commonly used CVD risk functions, including the FHS (Framingham Heart Study) functions for hard coronary heart disease (FHS CHD) and atherosclerotic CVD (FHS ASCVD) and the American College of Cardiology/American Heart Association (ACC/AHA) pooled cohorts equations (ACC/AHA ASCVD). To assess the functions in a cohort of HIV-infected men, we compared regression coefficients and calculated calibration and discrimination of the established functions. We hypothesized that the established CVD risk functions would underestimate risk in HIV-infected individuals.

Methods

The data, analytic methods, and study materials will be made available to other researchers for purposes of reproducing the results 1 year after the dataset is locked through Dataverse (https://dataverse.harvard.edu/).

FHS and ACC/AHA Functions

Three established CVD risk functions were evaluated: the FHS hard CHD function (FHS CHD),²⁰ the 2008 FHS global CVD function (FHS ASCVD),²¹ and the 2013 ACC/AHA function (ACC/AHA ASCVD).²⁵ The FHS global CVD function was developed for a composite CVD outcome and was modified for the present study to ASCVD using the algorithm described in the Outcomes section and in the original 2008 manuscript.¹⁹ Risk factors included in the FHS functions are age, diabetes mellitus, smoking, blood pressure category, total cholesterol, and high-density lipoprotein (HDL). Risk factors included in the ACC/AHA function are age, diabetes mellitus, smoking, systolic blood pressure, total cholesterol, HDL, and antihypertensive treatment. We adapted the 10-year risk prediction functions to 5-year functions to align with the study follow-up period. Given that the proportional hazards assumption was met for variables in the Cox models, we developed a 5-year risk model for each function by (1) applying the Cox regression coefficients from the 10-year risk models to the 5-year risk models, and (2) replacing in the model 10-year baseline survival (event-free) estimates at the average values of the risk factors with 5-year baseline survival (event-free) estimates at the average values of the risk factors.^19,20 The 5-year functions are included in the Appendix in the online-only Data Supplement.

HIV Cohort

The HIV population was identified from the Partners HIV cohort, an observational clinical care cohort of HIV-infected individuals receiving longitudinal HIV care through the Partners HealthCare System in Boston, Massachusetts. Data are derived from the Research Patient Data Registry, a data registry that includes comprehensive clinical information for all patients from the Partners HealthCare System. HIV-infected patients were initially identified from the Research Patient Data Registry by the presence of ≥1 International Classification of Diseases-Ninth Revision-Clinical Modification (ICD-9-CM) code for HIV (042 or V08). The HIV cohort was then limited to patients with ≥4 ICD-9-CM codes for HIV based on a validation study that demonstrated this definition of HIV status to be 88.9% sensitive and 98.6% specific for HIV infection based on physician medical record review. We limited the analyses to men because of low event numbers among women. Patients selected for inclusion were required to have ≥1 clinical encounter in calendar years 2006 to 2008, a blood pressure measurement available in 2006 to 2008, lipid laboratory values available in calendar years 2004 to 2008, smoking status available at baseline (2006–2008), and first HIV code that occurred before the start of observation for each individual. Exclusion criteria included history of a CVD event before the start of observation, <30 or >74 years of age for the FHS CHD and ASCVD functions, and <40 or >79 years of age for the ACC/AHA ASCVD function. The medical records of several prespecified subgroups in whom participants did not meet eligibility criteria by automated review were manually reviewed by a physician or research nurse to determine whether patients met inclusion criteria for the study; these included patients with exactly 3 ICD-9-CM codes for HIV, patients with the first HIV diagnosis after the observation start date, and patients missing 1 component of the CVD risk functions. The start of observation occurred on the date of risk score calculation, during a window period from January 1, 2006, to December 31, 2008. Participants who did not experience the outcome were censored at the earliest of noncardiovascular death, last follow-up visit, or 5 years after an individual’s start of observation. The study was approved by the Partners Human Research Committee, and no informed consent was required.

Calculation of Predicted CVD Risk

The baseline visit (visit of the risk score calculation) was the participant’s first visit that occurred between January 1, 2006, and December 31, 2008, that had nonmissing values for the risk factors necessary to generate a risk score. Age used was at baseline for each individual patient. Blood pressure data were obtained through electronic health record health maintenance–coded field data and by query of free text notes. Total and HDL cholesterol were obtained from laboratory data. Diabetes mellitus diagnosis was validated for this study and was defined as the presence of ≥3 ICD-9-CM codes of 250 and all subtypes (sensitivity, 86.2%; specificity, 93.0%). Smoking status was obtained through application of an natural language processing-based algorithm developed and validated in the Partners HIV cohort.²⁷ For the ACC/AHA function, white coefficients were used because this race/ethnicity was the most common and because there was no evidence of an interaction between race and traditional CVD risk factors on the outcomes, as discussed later.

Outcomes

Hard CHD was defined as MI (ICD-9-CM code 410 and all subtypes) or coronary death. ASCVD was defined as MI (ICD-9-CM code 410 and all subtypes), stroke (ICD-9-CM codes 433–434 and all subtypes), or coronary death. The 2008 FHS function is for global CVD. Before applying it to the HIV sample, the function was adapted for the outcome of ASCVD for this analysis by using the adjustment factor of the ratio of the number of patients in the cohort with ASCVD to the number of patients with global CVD, following the methodology as outlined in the 2008 FHS global CVD publication.²¹ Global CVD is defined as MI (ICD-9-CM code 410 and all subtypes), stroke (ICD-9-CM codes 433–434 and all subtypes), coronary death, coronary insufficiency (ICD-9-CM codes 411, 412, and 414, and all subtypes), angina (inpatient ICD-9-CM code 411.1x), transient ischemic attack (ICD-9-CM code 435 and all subtypes), peripheral artery disease (ICD-9-CM codes 440.2x, 440.3x, and 440.4x), or heart failure (ICD-9-CM code 428 and all subtypes). Outcomes were identified by the presence of an ICD code for the relevant outcome that occurred after the start of observation; only the first code was included. ICD-based ascertainment of outcome events was validated by reviewing a randomly selected subset of patients with and without ICD codes for MI and stroke. Medical record review by a trained clinical nurse was the gold standard. For MI, sensitivity was 81.5%, and specificity was 86.3%. For stroke, sensitivity was 86.7%, and specificity was 78.8%. Coronary death data are obtained from the Partners Research Patient Data Registry, data on inpatient deaths are provided by the hospital, and data on out-of-hospital deaths are obtained from the Social Security Administration’s Death Master File. The Death Master File does not contain cause of death, so for the present study, medical records for all patients who died were individually reviewed by a physician to determine whether the cause of death was coronary.

Statistical Analysis

Analyses described next were carried out separately for each of the 3 risk functions (FHS CHD, FHS ASCVD, and ACC/AHA ASCVD) and were limited to men because of low event numbers among women. Race/ethnicity groups were combined in the analyses because there were no significant interactions between race and the other risk factors on the outcomes at the 0.10 level of significance. We used white coefficients for the ACC/AHA function in the primary analysis. FHS functions were developed mostly on white participants, and there are no separate models for other races. In sensitivity analyses, we repeated all analyses (1) applying black coefficients for the ACC/AHA function to the entire HIV cohort, and (2) stratifying the analysis by race. For the race-stratified analyses, we categorized patients as black only, white only, or nonblack (white, Asian, Hispanic, other, or unknown) in separate analyses, using white coefficients for the white-only and nonblack groups and black coefficients for the black-only group.

Comparison of Coefficients

Cox proportional-hazards modeling was used to generate regression coefficients for each of the risk factors using data from the HIV cohort. Regression coefficients generated for the HIV cohort were descriptively compared to those previously generated for each function from FHS data. The coefficients generated from the HIV data were used to generate an HIV function that represents the best prediction function based on risk factor and outcome data from the HIV cohort, using the same traditional CVD risk factors as those included in the original functions and not including HIV-related risk factors.²⁰ We did not directly compare relative risks between the ACC/AHA ASCVD and HIV functions because of the difficulty of interpretation of relative risks for the interaction terms included in the ACC/AHA function.

In supplemental analyses, we generated a new model based on risk factor and outcome data from the HIV cohort for men and women combined. For each function, we included all original risk factors and interaction terms with sex for each risk factor. We then used backward selection to remove nonsignificant interaction terms 1 at a time using a 0.10 level of significance (the interaction term with the largest P value >0.10 was removed in the backward approach, and we reran the regressions; the process was repeated until all remaining interaction terms had P<0.10). Original risk factors were retained in the model regardless of their significance. Analyses were conducted using SAS version 9.4.

Discrimination

To assess the ability of each prediction function to separate individuals who experience an event from individuals who do not, we calculated the c statistic. For each outcome, 2 c statistics were calculated. The first applied the established function (FHS CHD, ACC/AHA ASCVD, or FHS ASCVD) to the HIV data, and the second applied the HIV function, developed using risk factor and outcome data from the HIV cohort rather than using FHS or ACC/AHA cohort data, to the HIV data.

Calibration

For each of the 3 risk functions and for the HIV function, calibration (goodness of fit of predicted risk to observed risk) was assessed using the Demler et al modification²⁸ of the D’Agostino and Nam²⁹ χ² statistic. The HIV cohort was divided into deciles of predicted 5-year risk of CVD according to each risk function. For calculation of the χ² statistic, we collapsed deciles with small numbers of categories until all resulting groups contained ≥5 events as required by Demler et al. This strategy was used to ensure a stable χ² statistic. For each resulting group, the mean predicted risk and the observed Kaplan–Meier estimate of event rate were calculated and compared using the Demler/D’Agostino-Nam χ² statistic; a nonsignificant χ² (P value >0.05) indicates good calibration (ie, good fit of predicted risk to observed risk). In the event of poor calibration of the published risk models to the HIV data, recalibration was performed by replacing the mean values of risk factors and average incidence rates used from the established functions with those derived from the HIV cohort as discussed in D’Agostino et al.²⁰ Calibration was then recalculated in a similar manner where participants were grouped in the following 3 categories of predicted risk instead of deciles: <5%, 5 to 7.5%, >7.5%.

Results

Baseline Characteristics

Table 1 shows demographic and clinical characteristics of the HIV cohort among men 30 to 74 years of age for all individuals and according to CHD event status. The cohort was comprised of 1272 men, with median follow-up time of 4.4 years. Mean HDL was 43.7 mg/dL among patients without CHD events versus 40.7 mg/dL among those with events, and the proportion of patients with optimal or normal blood pressure was 60.9% among patients without CHD events versus 54.2% among those with events. More than 45% of patients were smokers; among patients with CHD events, 60.4% were smokers (versus 44.9% among those without events). Diabetes mellitus was present in 13.5%, with a rate of 27.1% among patients with CHD events (versus 13.0% among patients without events). Among all patients, 94.6% were receiving antiretroviral therapy. Median CD4 cell count was 457; among patients without CHD events, median CD4 count was 457 (versus 348 among those with events). The rate of viral suppression was 77.1% overall, 77.5% among patients without CHD events, and 66.7% among patients with CHD events.

The 5-year hard CHD event rate was 3.8% (48/1272), and the 5-year ASCVD event rate was 6.1% (78/1272). Incidence rates for hard CHD and ASCVD are shown in Table 1. Median risk scores for each function are shown in Table I in the online-only Data Supplement and include predicted risk scores based on established functions, observed risk, and predicted risk scores based on the Cox models developed from the HIV cohort (HIV function).

Table II in the online-only Data Supplement shows demographic and clinical characteristics of the HIV cohort among women 30 to 74 years of age overall and according to CHD event status.

Comparison of Coefficients

Table 2 shows coefficients and 2-sided 95% confidence intervals for risk factors from the HIV cohort’s regression model and also displays published coefficients derived from the original cohort’s (FHS or ACC/AHA) model for each function among men. Qualitative comparison of coefficients was performed, taking into consideration that interpretation of results is limited by a low number of events. The majority of the coefficients developed from the HIV function did not achieve statistical significance (confidence interval of the coefficient contained 0) for each of the 3 established functions evaluated. For hard CHD, increasing total cholesterol category was associated with higher hard CHD risk in both the FHS and HIV-developed models. Being in the lowest HDL category (<35) yielded the largest hard CHD risk, and being in the highest HDL category (≥60) was associated with the lowest risk in both hard CHD models. Hypertension was associated with a higher risk of hard CHD in both models. As anticipated, age, diabetes mellitus, and smoking were associated with higher hard CHD risk in both models.

For the ACC/AHA ASCVD function, age, total cholesterol, systolic blood pressure, smoking, and diabetes mellitus were associated with higher ASCVD risk in both the ACC/AHA and HIV models. For both models, as expected, increased HDL was associated with lower ASCVD risk.

For the FHS ASCVD function, all risk factors had similar relative effects in the FHS and HIV models, with increased age, total cholesterol, systolic blood pressure, smoking, and diabetes mellitus associated with higher risk and HDL associated with lower risk.

Discrimination

Table 3 shows the c statistic, a measurement of discrimination, and its 2-sided 95% confidence interval for each function applied to the HIV cohort. In the table, the Original Function rows indicate the c statistic derived from applying the FHS CHD, ACC/AHA ASCVD, or FHS ASCVD function to the HIV cohort’s data, respectively. The HIV Function row indicates the c statistic derived from a Cox regression model developed from the HIV cohort using the same variables as contained in the relevant FHS or ACC/AHA function but using risk factor values and outcome data measured on the HIV cohort (HIV function). Discrimination of all published functions when applied to the HIV cohort was suboptimal (Original function), with c statistics of 0.68 for FHS CHD, 0.65 for ACC/AHA ASCVD, and 0.67 for FHS ASCVD. Application of the Cox regression model developed from the HIV cohort to the HIV cohort (HIV function) resulted in improved but not optimized discrimination for most groups (c statistics of 0.73 for FHS CHD, 0.66 for ACC/AHA ASCVD, and 0.67 for FHS ASCVD).

Calibration and Observed Versus Predicted Risk

Figure 1 shows observed and predicted 5-year risk for each of the functions by decile of predicted risk for all participants. For all functions, observed risk exceeded predicted risk for most deciles of predicted risk. Figure 2 shows observed and predicted 5-year risk for each of the functions by predicted risk group (<5%, 5%–7.5%, and >7.5%). Observed risk exceeded predicted risk for all categories in all functions except for >7.5% predicted risk for the FHS hard CHD function.

• Download figure
• Download PowerPoint

• Download figure
• Download PowerPoint

Table I in the online-only Data Supplement shows median risk scores by function and compares predicted versus observed risk. For all 3 functions, observed risk exceeded predicted risk, whether predicted risk was calculated using the traditional functions or the HIV function. All predicted risk scores, however, were greater and closer to the observed risk when the HIV function was applied versus when the traditional functions were applied.

The calibration, or goodness of fit, of the FHS and ACC/AHA functions applied to the HIV cohort was assessed using the calibration χ² statistic. For the 3 functions assessed, calibration was poor and reflects inadequate fit of the general population functions to the HIV cohort. For the FHS CHD function, the calibration χ² statistic was 13.6 (P=0.019, 5 degrees of freedom). For the ACC/AHA function, the calibration χ² statistic was 23.9 (P=0.001, 7 degrees of freedom). For the FHS ASCVD function, the calibration χ² statistic was 24.6 (P=0.0004, 6 degrees of freedom).

We recalibrated the FHS or ACC/AHA model to attempt to improve the model fit of these functions by using baseline survival and mean risk factor values from the HIV cohort instead of the FHS or ACC/AHA cohorts’ values. After recalibration, goodness of fit remained poor for all functions, and model performance did not improve (data not shown).

In the primary analysis for the ACC/AHA ASCVD function, white coefficients were used given that there was no evidence of any interaction between race and traditional CVD risk factors in the HIV cohort. We repeated all analyses using black coefficients for the ACC/AHA function and generated similar results. To further confirm that each function poorly discriminated and underestimated risk in the HIV cohort, we conducted analyses stratified by race and showed that discrimination remained moderate and calibration remained poor (Tables III and IV in the online-only Data Supplement).

In supplemental analyses, we generated a new model (HIV function) among men and women combined, including significant interaction terms with sex for each risk factor. Results of this analysis are shown in Table V in the online-only Data Supplement.

Discussion

We evaluated 3 established cardiovascular risk prediction functions in an HIV cohort and found that the functions systematically underestimate CVD risk in HIV. Discrimination and calibration were both suboptimal when applying the functions to a cohort of largely antiretroviral therapy–treated men engaged in HIV care, indicating that risk prediction algorithms developed for use in the general population may underperform and may be inadequate in their ability to appropriately order CVD risk and accurately calculate predicted risk. Our findings suggest that established CVD risk functions do not provide an accurate estimation of risk in the setting of HIV disease and may fail to identify patients at elevated CVD risk who would benefit from aggressive risk reduction. Adaptation of current CVD risk functions with the incorporation of new risk factors is likely to be needed for safe and accurate risk estimation in HIV.

The question of transportability of CVD risk prediction functions to populations that differ from those in which they are developed has received considerable attention.²⁰ To date, the FHS functions have been the most widely used to quantify the absolute risk of developing a specific CVD outcome for an individual patient over a given time period.^20,30 The release of ACC/AHA guidelines on CVD risk assessment and cholesterol treatment represents a turning point in the approach to CVD prevention in the general population,^25,26,31 with the development of a new CVD risk prediction function derived from a more representative population and utilizing the broader outcome of ASCVD. The applicability of established functions in the setting of HIV, however, remains incompletely understood. The ACC/AHA guidelines specifically call for validation in diverse settings,²⁵ and in this analysis, we find established prediction functions to be inadequate when applied to individuals with HIV.

Developing accurate CVD risk functions in HIV-infected individuals has been identified as a priority by both HIV and cardiovascular advocacy groups.^25,26,32 The hypothesis that conventional CVD risk prediction functions may not perform accurately in HIV is based on proposed mechanistic factors. Although traditional CVD risk factors are prevalent in HIV and contribute to CVD risk,⁶ extensive data now suggest that HIV-specific inflammation and immune activation play prominent roles in mediating CVD risk.^11,13,14 Even in virally suppressed patients, inflammation and immune activation continue to be abnormal,³³ and persistent chronic inflammation may increase long-term CVD risk and severity.^34,35 Adding to the complexity of the mechanism of HIV-associated CVD, several individual HIV medications have been shown to be associated with increased CVD risk,^16,17 although whether they mediate risk via effects on traditional risk factors versus novel mechanisms remains unclear and is likely to differ by drug. Established CVD risk algorithms were modeled after risk factors common in the general population and do not reflect these unique mechanistic factors thought to drive HIV-related CVD.

Prior studies in HIV have evaluated the correlation between CVD risk scores and subclinical atherosclerosis³⁶ and suggest that FHS functions underestimate the risk of acute MI³⁷ and stroke.³⁸ Data from the HOPS (HIV Outpatient Study)³⁹ demonstrated accurate risk estimation but suboptimal discrimination for the FHS global CVD function, underestimation and reasonable discrimination by the ACC/AHA and D:A:D (Data collection on Adverse Events of Anti-HIV Drugs) functions,^40,41 and overestimation with poor discrimination for the SCORE (Systematic Coronary Risk Evaluation) function.⁴² A recent study from the Centers for AIDS Research Network of Integrated Clinical Systems cohort found the ACC/AHA function to adequately discriminate MI risk and to be moderately calibrated, with variability in performance by demographic subgroups.⁴³ Additionally, incorporation of HIV-specific variables did not improve the risk score.⁴³ Our study builds on prior knowledge by providing a formal assessment of several established functions’ performance in terms of discrimination, calibration, and comparison of regression coefficients and provides new knowledge by evaluating the ACC/AHA and 2 FHS functions in a cohort of HIV-infected individuals with the ability to evaluate the outcome of ASCVD, including MI and stroke.

Our data support the hypothesis that established CVD risk functions underestimate risk for HIV-infected men. Our assessment of the models’ discriminatory capacity indicated that all the functions suboptimally ordered risk in the HIV cohort; the c statistics, which represent the probability that patients who develop the outcome will be assigned a higher risk than those who do not, were lower than the standard threshold for all 3 functions when applied to the HIV cohort. Observed risk was greater than predicted risk across all 3 risk functions and for most deciles and groups of predicted risk. The degree of agreement between predicted and observed risk was formally studied through assessment of calibration, with large χ² statistics and significant P values indicating poor model fit for all 3 functions. Comparison of regression coefficients, applying the original function versus the HIV function, indicates different magnitudes and, at times, directions of the effect of traditional risk factors on cardiovascular outcomes, although qualitative comparisons are limited by a relatively low number of events.

To attempt to improve model performance, we generated an HIV function, which represents the best function that can be developed based on risk factor and outcome data from the HIV cohort but using risk factors identical to those included in the established function. We evaluated these best HIV functions—developed from the 3 established functions—in terms of traditional risk factors, discrimination, and calibration. For these newly developed functions, we found that few of the regression coefficients for traditional CVD risk factors achieved significance. Discrimination, as indicated by the c statistic, improved for the HIV functions but remained inadequate overall, and recalibration only modestly improved model fit for each HIV function. These findings indicate that the best model we can develop, directly computed from the HIV data, only marginally improves risk prediction, and that additional research is needed—likely including the addition of new variables—to develop an adequate risk model in the HIV population.

Together these data suggest that cardiovascular risk prediction using established algorithms developed for the general population remains suboptimal in the setting of HIV disease, and that as a first step, the discriminatory capacity of the models needs to be improved. Our findings of the models’ moderate to poor discriminatory capacity are consistent with the possibility of an unaccounted-for risk factor that is relevant to HIV-infected individuals but not to the populations from which the functions were developed. This missing variable is likely to be related to HIV-associated inflammation and immune activation, which have been shown to directly impact cardiovascular risk in this population.^44–46 Although HIV-related, novel CVD risk factors have been demonstrated to be significantly associated with CVD outcomes, it is unknown whether the incorporation of such factors into established risk prediction functions will improve the performance of the functions. Moreover, risk factors considered for inclusion should be clinically relevant to facilitate widespread implementation of a new HIV-specific function. Although the primary purpose of this study was to investigate the role of previously studied traditional CVD risk factors in risk prediction in HIV, future work is planned to evaluate the incorporation of HIV-specific variables, including HIV disease parameters and medications as well as HIV itself, into tailored risk prediction algorithms.

Our study has several limitations, including the possibility of inaccurate outcome ascertainment because of missing data or incorrect classification. CVD outcomes were validated for the current study; as 10-year outcome data become available, adjudication of events is planned. The follow-up time of 5 years (versus 10 years) was selected to balance and optimize completeness and accuracy of electronic health record data (which has improved over time in the Research Patient Data Registry) with longer duration of follow-up. Although the lower number of events limits interpretation of current results, we are prospectively collecting 10-year data and anticipate an increasing number of events as more follow-up time is accumulated. Additionally, we did not include women in the analysis because of the difficulty of interpreting results because of low event numbers. Thus, findings are generalizable to HIV-infected men. Future analyses of 10-year events including women are planned.

Conclusions

In conclusion, cardiovascular risk prediction using established general population algorithms was found to be inaccurate in HIV infection, with consistent underestimation of risk. In this study using a longitudinal HIV clinical care cohort, we assessed the performance of 2 established FHS functions and the ACC/AHA function and found that all functions resulted in poor model fit when applied to the HIV population. Our findings suggest that current application of established CVD risk functions in clinical practice may not result in accurate CVD risk assessment, and that reliance on such measures may fail to identify high-risk HIV-infected individuals who would benefit from aggressive cardiovascular risk modification. Refining current CVD risk functions through the addition of novel HIV-related factors, including HIV, may improve CVD risk function performance. The lack of transportability of established CVD functions to HIV underscores the need for tailored CVD risk prediction strategies for this at-risk population to manage and prevent one of the most serious complications of long-term HIV infection.

Acknowledgments

The authors are grateful to the Partners HealthCare Research Patient Data Registry group for facilitating use of their database and to Steven McDermott, RN, for assistance with validation studies.

Footnotes