Performance of Comorbidity Measures to Predict Stroke and Death in a Community-Dwelling, Hypertensive Medicaid Population

Controlling for differences in comorbidity is particularly important in epidemiological, outcome, and health services research. However, because there is no gold standard for assessing comorbidity, investigators validate alternative measures by determining their ability to predict important health outcomes such as mortality, healthcare costs, and quality of life.¹ Because the predictive performance of alternative comorbidity measures depends in part on the outcome of interest, investigators should choose comorbidity indices appropriate to both their study population and outcome.^2,3 Choice among alternative comorbidity measures is also typically guided by convenience and the availability of relevant data. Therefore, the most widely used and studied comorbidity indices use billing data,^1,3,4 because these administrative data can readily assess real-world inpatient and outpatient diagnoses, healthcare utilization experience, and medication use. Although comorbidity indices are not yet widely used by clinicians to assess risk for individual patients, they have great potential for helping practitioners identify patients at highest risk of premature disability and death.

Although many alternative approaches to assessing comorbidity using administrative data have been validated,^1,3,4 the Charlson Index, a disease severity weighted index of comorbidity, is perhaps best studied, validated, and widely used. Charlson and colleagues first developed the index weightings using chart review data collected from an index hospitalization to predict 1-year survival. They validated the index on an independent population of patients with breast cancer.⁵

Subsequently, numerous researchers have adapted the Charlson Index for use with administrative data. Schneeweis and others have shown that comorbidity scores based on both outpatient and inpatient diagnostic codes are similarly predictive of major outcomes.^6–9 However, the predictive performance of the Charlson Index using outpatient and inpatient data has rarely been assessed.¹⁰ Recent studies suggest that the longer list of comorbidities proposed by Elixhauser may have improved predictive performance.^1,11–13 These comorbidity assessment approaches have not been adequately compared nor have they been specifically validated for use in longitudinal hypertension studies.^1,11–13 Additionally, many authors have demonstrated similar predictive performance using simple measures of overall healthcare utilization and medication use,⁴ and some suggest that more disease-specific risk factor assessment approaches focusing on selected comorbidities may be more appropriate in cardiovascular epidemiological studies.¹⁴

This study aims to determine whether the Charlson Index is a valid index of comorbidity for community-dwelling hypertensive Medicaid enrollees and whether the Charlson Index is a better indicator of comorbidity than approaches using multiple independent comorbidities or other simpler indices relying solely on levels of healthcare or medication use. In addition, the authors wanted to determine if the Charlson Index predicts stroke and if it performs better than assessing selected comorbidities particularly known to be associated with stroke. Finally, given the increasing availability of real-time administrative data through electronic claims submission, the authors wanted to see if comorbidity measures calculated for a period immediately before an event would better predict stroke and death than the same measures when calculated for a baseline period that was longer but more removed in time from the event of interest.

Materials and Methods

The study population consisted of chronic medication-treated hypertensive individuals enrolled in Tennessee’s statewide Medicaid managed care program (TennCare) for at least 3 continuous years from 1994 through 2000 inclusive. A longitudinal retrospective cohort study design was used. The hypertensive cohort was defined using inpatient and outpatient administrative data. To select chronic medication-treated hypertensives, the cohort included all patients with (1) continuous TennCare eligibility for at least 3 years of the 7-year study period; (2) aged 18 to 65 in each study year; (3) yearly diagnosis of hypertension defined as one or more inpatient or professional (inpatient or outpatient physician service) claims with an International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) code of 401.x in each of 2 baseline years; and (4) yearly receipt of at least one antihypertensive medication prescription for the 2 baseline years.

The definition for hypertension used is based on claims-based definitions shown in multiple studies to have high validity for the diagnosis of hypertension. These studies have demonstrated that the use of both diagnosis and prescription claims and multiple years of data result in higher diagnostic specificity.^15,16

Patients who experienced stroke or death during their baseline 2-year period were excluded. For this purpose, stroke was defined as any inpatient or professional claims including an ICD-9-CM diagnosis code 430 to 438. This eliminated virtually 100% of those with s history of stroke.^17–19 After the 2-year baseline period, the cohort of eligible patients (n=49 479) was followed for a 1- to 5-year follow-up period in which study outcomes could occur.

Tennessee Medicaid encounter files served as the primary data sources for the study. These files include all paid claims for inpatient, professional, and pharmacy services for enrollees using Clinical Procedural Terminology codes. Five diagnosis fields using ICD-9-CM codes were included in all inpatient and professional claims. The pharmacy database has been well validated.²⁰ For the period of the study, approximately 25% (1.2 to 1.4 million) of Tennessee’s residents were enrolled in TennCare. Demographic variables were assessed using baseline period administrative eligibility and demographic files, and comorbidity was assessed using baseline and follow-up period inpatient, professional, and pharmacy administrative data for paid claims.

Demographic Variables and Clinical Outcomes

Demographic variables acted as control variables and included age, gender, race, disability status, uninsurable eligibility status, region, urban/rural status, managed care organization type, managed care organization for profit status, and median income of county residence. Dependent variables were the 2 study clinical outcomes: time to death and time to stroke. Demographic variables were included as control variables in models assessing various comorbidity measures that were significant in preliminary Cox models for predicting each study outcome as described in the “Data Analysis” section.

Death, the primary clinical outcome of the study, was defined as all-cause mortality according to Tennessee Department of Health Vital Records. Because the linkage of Vital Records and TennCare administrative data used Social Security number, a field known to be highly accurate in both databases, the validity of the linkage approached 100%. Cohort members were considered to have experienced an incident stroke if they had any inpatient claims including a primary ICD-9-CM diagnosis of subarachnoid or intracerebral hemorrhage (codes 430 to 431), any diagnosis of cerebral infarction (codes 433.11, 433.21, 433.81, 433.91, 434.01, 434.11, and 434.91), or any diagnosis of acute, but ill-defined, cerebrovascular disease (code 436) during their follow-up period. In addition, cases were classified as negative for stroke following the method of Tirschwell if diagnoses of traumatic brain injury (codes 800 to 804, 850 to 854) were present. These stringent requirements for stroke diagnosis followed the recommendations of Reker and others and were designed to maximize the positive predictive value and specificity of the stroke diagnosis.^{17,19,21–23}

Comorbidity Measures

Comorbid conditions were assessed using alternative approaches that were suggested in the literature as appropriate for chronic hypertensive individuals. The 7 approaches to assessing comorbidity that were considered included: (1) Modified Charlson Index; (2) Charlson comorbidities; (3) Elixhauser comorbidities; (4) prior major cardiovascular disease event (CVD); (5) traditional risk factors for cerebrovascular accident (CVA); (6) overall healthcare utilization (HCU)); and (7) antihypertensive medication utilization (MU). For the diagnosis-based comorbidity assessment approaches, comorbidities were defined as shown in supplemental Tables I and II, available online at http://stroke.ahajournals.org. Comorbidity measures were calculated for each participant both for the baseline 2-year period and for a 3-month or 6-month period before occurrence of the study outcome of interest or study conclusion.

Modified Charlson Index and Charlson Comorbidities

A modified version of the Charlson Index was used based on the Romano adaptation and following Schneeweiss’ method for adding ambulatory diagnoses.^4,7,9,24 The authors also followed the Goldstein²⁵ adaptation and eliminated from consideration 2 of the original 17 diagnostic categories (eg, cerebrovascular disease, paralysis) because these categories are reflected in the condition being evaluated. All updated ICD-9-CM codes for the Charlson Index in use for the period of the study were included in assessing for the Charlson comorbidities¹³ as shown in supplemental Table I. The Modified Charlson Index score was calculated for the baseline period and for every 6 months in the follow-up period before outcome or study conclusion (not including the 6-month period during which the outcome occurred). Using the same methodology, all 15 unweighted individual Charlson comorbidities were assessed for each subject for the baseline and the follow-up periods.

Elixhauser Comorbidities

Following the Elixhauser definitions, 28 unweighted individual comorbidities were similarly assessed for each subject for the baseline and the follow-up periods, excluding 2 comorbidities reflected in the condition being evaluated (CVD and paralysis)^1,11–13 as shown in supplemental Table I. These 28 variables were only assessed individually because the Elixhauser method does not have a weighted scoring system. The original Elixhauser approach was modified by: (1) including ambulatory diagnoses; and (2) omitting an initial Diagnosis Related Groups screen as other authors have done,^11–13 because it is not applicable to ambulatory data.

Cardiovascular Event (cardiovascular disease)

Prior major CVD event represents a selected group of serious cardiovascular comorbidities for which hypertensive individuals are particularly at risk.²⁶ Numerous independent hypertension clinical trials have demonstrated that history of these CVD comorbidities places hypertensive individuals at high risk of stroke and death.²⁷ This comorbidity measure was assessed following the standard definition used in clinical trials of hypertension treatment²⁸ and included history of a diagnosis of nonfatal myocardial infarction, congestive heart failure (CHF), and/or coronary artery bypass graft/percutaneous transluminal coronary angioplasty as shown in supplemental Table II. Administrative data accurately reflect CVD events.¹ CVD event was assessed for the baseline period and every 6 months of the follow-up period before the outcome or conclusion of the study.

Traditional Stroke Risk Factors

The CVA comorbidity model included 4 dichotomous variables assessing the major traditional stroke risk factors, including CHF, atrial fibrillation, transient ischemic attack, and myocardial infarction ²⁹ as shown in supplemental Table II. Although many studies have shown that inpatient discharge codes are specific for CHF,³⁰ Goff in particular shows that use of inpatient codes alone results in substantial underreporting.³¹ Go and colleagues have shown that 2 ICD-9-CM outpatient codes accurately assess presence of atrial fibrillation.³² Hill and colleagues have clearly shown that emergency room diagnosis of transient ischemic attack according to administrative data is a strong risk factor for stroke.³³ CVA variables were assessed for the baseline period and every 6 months in the follow-up period before the outcome or conclusion of the study.

Healthcare Utilization

Marini et al have shown that overall HCU is a marker of comorbidity and may predict clinical outcomes better than some diagnosis-based scores.³⁴ The 4 independent interval variables in the HCU comorbidity model included: outpatient visit frequency, emergency visit frequency, hospital visit frequency, and number of hospital days. HCU variables were assessed for the baseline period and every 3 months in the follow-up period before the outcome or the conclusion of the study.

Antihypertensive Medication Utilization

Levels of antihypertensive MU are known to be associated with both higher severity of illness and level of comorbidity because these medications are used to treat a variety of cardiovascular and other conditions in addition to hypertension.³⁵ Many studies have looked at overall prescription drug utilization as an indicator of comorbidity, but this study focused on antihypertensive medication use because this may better reflect comorbidity in the hypertensive population.³⁶ The antihypertensive MU measure indicates the number of unique antihypertensive medication classes for which prescriptions were filled in the period. MU was calculated as an interval variable with a range from 1 to 9. MU was assessed for the baseline period and every 6 months in the follow-up period before the outcome or conclusion of the study.

Data Analysis

The 7 comorbidity assessment approaches were compared by developing Cox models for each approach with both baseline and follow-up data and for both the stroke and death outcomes.³⁷ This yielded a total of 28 alternative Cox models for predicting the outcomes. In all the models, demographic variables were included as control variables that were first found to be significant in initial Cox models that only included demographic variables. Thus, models for stroke included the same 4 demographic variables (age, race, disability status, and urban/rural status), and those for the death outcome included the same 9 demographic variables (age, gender, race, disability status, uninsurable eligibility status, median income of county of residence, region, urban/rural status, and managed care organization type).

The 14 models for each outcome were compared using Akaike’s information criterion (AIC) model fit statistic.^38,39 AIC scores were calculated for each model. The magnitude of AIC has little meaning. Rather the focus is on the relative size, and the model with the smaller AIC is preferred.

where L=likelihood function, θ=a set of regression coefficients, n=sample size, and K=the number of variables in the model.

In the current study, sample size was 49 479. The numbers of regression coefficients ranged from 5 to 32 for each of the stroke models and from 10 to 37 for each of the death models. AIC differences were then calculated following the method of Anderson and Burnham to select the best approximating model.⁴⁰ AIC difference (ΔAIC), which is the difference between the AIC and the minimum of AIC over all candidate models, provides a quantitative measure of model plausibility. A zero difference indicates the best model, models having ΔAIC <2 to 3 are nearly tied, models having ΔAIC between 7 and 10 are considered fair, and models having ΔAIC >7 to 10 are substantially inferior.⁴⁰ All statistical analyses were performed using SAS version 9.1.⁴¹

Results

The 49 479 subjects who met full study criteria were followed for a mean of 2.66 years (range, 0.005 to 5 years) after the 2-year baseline period. The cohort participants had a mean age of 48.5 years (range, 20 to 64 years) at the end of the baseline period. A total of 67.7% were female and 30.0% were black. During the follow-up period, there were 619 strokes and 2051 deaths, yielding a 1.3% risk of stroke and a 4.1% risk of death for cohort participants during follow-up.

Baseline Modified Charlson Index scores ranged from 0 to 16 with a mean of 1.7 and a median of 1, consistent with a highly skewed distribution (Figure). The frequencies of the various component diagnostic codes are shown in supplemental Tables I and II. Of note, myocardial infarction (9.4%), CHF (12.1%), chronic obstructive pulmonary disease (22.6%), and diabetes (21.3%) were quite prevalent in this population at baseline. Results of Cox models using baseline or follow-up data for both outcomes revealed that all of the comorbidity approaches were predictive (Table 1).

• Download figure
• Download PowerPoint

Risk of stroke increased significantly with increasing baseline and follow-up indicators of comorbidity. Table 1 shows that when baseline data were used, the Charlson comorbidities model had the smallest AIC value for the stroke outcome. For all of the alternative comorbidity models for stroke using baseline data, ΔAIC was >10 indicating that all were substantially inferior. The Elixhauser comorbidities model had the smallest AIC value for the stroke outcome when follow-up data were used. All of the other models using follow-up data to predict risk of stroke were substantially inferior except the Charlson comorbidities for which ΔAIC was 6.883 indicating good fit.

Risk of death also increased significantly with increasing baseline and follow-up indicators of comorbidity. Table 1 also shows that when baseline data were used, the Charlson comorbidities model had the smallest AIC value for the death outcome. For all of the alternative comorbidity models for death using baseline data, ΔAIC was greater than 10 indicating that all were substantially inferior. The Elixhauser comorbidities model had the smallest AIC value for the death outcome when follow-up data were used. All of the other models using follow-up data were substantially inferior.

The Modified Charlson Index also predicted stroke and death, but alternative models were more robust. Modified Charlson Index models using both baseline and follow-up data were significant at P<0.0001, but the models using follow-up data were associated with higher hazards of stroke and death, indicating that Modified Charlson Index performs better when follow-up data are used.

Discussion

This study indicates that Modified Charlson Index, Charlson comorbidities, and Elixhauser comorbidities are all valid comorbidity assessment methods for predicting stroke and death in community-dwelling Medicaid enrollees with chronic medication-treated hypertension. It validates the Charlson and Elixhauser methods for use in longitudinal hypertension studies and adds to a growing group of validation studies showing that methods using outpatient as well as inpatient diagnoses accurately assess comorbidity and can be used to adjust for comorbidity in outpatient cohort studies of patients with chronic illnesses.¹⁰ The Charlson and Elixhauser methods using multiple unweighted comorbidities substantially outperformed several alternative suggested approaches for assessing comorbidity in patients with chronic hypertension. This study demonstrates that the Charlson and Elixhauser methods are generally better indicators of comorbidity than simpler approaches. The Charlson and Elixhauser methods performed similarly to each other with the better method dependent on the data used.

Investigators may still prefer to use Modified Charlson Index given its simplicity and the loss of power associated with adding 15 Charlson comorbidity variables or 28 Elixhauser comorbidity variables to predictive models in which comorbidity is not the major focus. The Modified Charlson Index still performs better than merely assessing overall levels of baseline healthcare utilization or assessing selected comorbidities particularly known to be associated with stroke.

Most of the previous studies using Charlson and Elixhauser comorbidities have focused on elderly populations with greater burden of chronic illness.^{7,10,25,30,42,43} This study showed that the Modified Charlson Index has similar predictive ability for younger Medicaid enrollees with substantially lower levels of comorbidity than in elderly Medicare populations.^24,44 For example, 57% of the current study’s population had a Charlson comorbidity score of zero or one compared with 29% reported by van Doorn et al.⁴²

This study also adds important information about the best timing of comorbidity assessment. It was shown that the Charlson comorbidities perform better than other models at predicting mortality when baseline data are used, and Elixhauser comorbidities perform better when follow-up data are used. Comorbidity models using follow-up data generally were associated with higher hazards of stroke and death than when baseline data were used. These findings suggest that investigators with the ability to use time-dependent data to calculate comorbidities should do so.

The AIC model fit statistic has several major advantages for comparing comorbidity adjustment approaches using time-dependent data. Goodness of fit is usually assessed using −2 log likelihood, but this approach does not adjust for the number of variables in the model, making it inappropriate in this study. Because this study used time-dependent covariates and assessed time to event using Cox models, the authors were not able to use the c-statistic to assess goodness of fit as previous investigators have done.⁴⁵ The AIC is particularly well suited for identifying the best model within the same candidate family, where differences among models are small.

Cox modeling has, as a major advantage, the ability to use all the information available in longitudinal studies and to assess impact of multiple covariates on time-dependent outcomes. The current study used a 7-year retrospective cohort database to compare alternative comorbidity assessment methods using baseline and time-dependent data. The study shows that models using time-dependent data generally performed better at predicting survival. Certainly using only baseline data in longitudinal epidemiological studies is acceptable, but this approach is an inefficient use of data. The current study documents small but significant improvements in modeling capability using the full range of data available.

One potential limitation of the AIC statistic is the inability to compare models using baseline data with models using follow-up data. However, the size of the hazard ratios can be used as a measure of predictive power. Although the AIC model fit statistic is an appropriate technique for model selection using time-dependent data, its ultimate disadvantage is that this approach does not allow the investigator to easily ascertain the variance explained by each model using an approach such as the receiver operating characteristics curve.⁴⁶ Future research needs to focus on the development of similar methods for constructing receiver operating characteristics curves using epidemiological studies that assess time to event using Cox modeling. Although several investigators have begun to develop these techniques,⁴⁶ these methods have primarily been applied in small studies with limited independent variables and have not been developed for application for large studies using multiple time-dependent covariates.

Another limitation of this study is that its results apply mainly to similar populations of middle-aged hypertensive individuals enrolled in Medicaid. For example, the traditional cerebrovascular accident risk factors model that performed well in predicting stroke may emphasize a cardioembolic mechanism of stroke that is more important in a younger hypertensive population. Clearly, different results might be found in different clinical populations. Further studies are needed to replicate these findings in other states, Medicaid populations, and clinical subgroups.

This study clearly documents that the Charlson and Elixhauser methods are valid measures of comorbidity for outpatient cohorts of chronic hypertensives, but it goes further to demonstrate that Charlson comorbidities perform better when baseline data are used, whereas the Elixhauser comorbidities perform better for short follow-up periods when comorbidities are assessed proximal to events of interest. Based on practical considerations, investigators may still prefer the Modified Charlson Index to either of the more complex approaches, and this study validates its use in outpatient hypertension studies. Further efforts are needed to make these research measures accessible to clinicians to assist them in identifying patients at highest risk of premature disability and death.

Footnotes