Prediction of Major Vascular Events in Patients With Transient Ischemic Attack or Ischemic Stroke: A Comparison of 7 Models

We compared the external validity (ie, calibration and discrimination) of 3 population-based models (the Framingham risk score for cardiovascular events,³ the Framingham risk score for stroke,⁸ SCORE,⁹ and the INDIANA project cardiovascular risk score¹⁰) and 4 stroke cohort-based models (Stroke Prognosis Instrument [SPI] II,¹¹ the Oxford TIA,⁷ the Dutch TIA study,¹² and the ABCD² score¹³) in an independent cohort of patients with a recent TIA or minor stroke (Table 1). All models were designed to predict long-term outcomes, except for the ABCD² score, which predicts 90-day risk of stroke. The stroke population-based models were designed to predict events after TIA or stroke; the population-based models were designed to predict first-ever stroke or cardiovascular events. These models were based on well-described large cohorts of patients with TIA or minor stroke and had the relative risks of the predictors estimated in a multiple (logistic or proportional hazards) regression model. The models predicted stroke and other cardiovascular events.

The SPI-I was developed in a small cohort of patients with a recent stroke. Kernan et al¹¹ created SPI-II by incorporating new predictive variables identified in a cohort of patients from the WEST study and validated it in 3 cohorts of patients from several secondary prevention trials.

Hankey et al⁷ developed a prediction equation for recurrent vascular events based on 8 clinical prognostic factors from a cohort of 469 hospital-referred TIA patients. The prediction model was validated in cohorts of TIA patients from the Oxfordshire Community Stroke Project and the UK-TIA study.¹⁴

The Dutch TIA trial investigators developed a set of predictors of major vascular events and of stroke. Their prediction model was based on data of patients with a TIA or minor stroke who entered a multicenter, randomized, controlled clinical trial with a 2×2 factorial design of high-dose vs low-dose aspirin and propanolol vs placebo.^12,15

The ABCD² score calculates the 2-, 7-, and 90-day stroke risk in TIA patients.¹³ This prediction model was based on TIA patients presenting within 1 day of onset of symptoms. We used the 90-day risk score because this was closest to the 2-year risk calculated in the validation cohort.

The 3 other models were population-based models that have been widely used to assess cardiovascular risk. For risk of major vascular events, we used the Framingham risk score for cardiovascular events published in 1998.³ This model is based on categorical variables. For risk of stroke, we used the Framingham risk score for stroke published in 1994.⁸

The SCORE prediction model, methodology differs slightly from the other models. Annual age- and gender-dependent rates of coronary and vascular event were estimated from the data in an accelerated failure time model (Weibull). This model made use of age as a measure of exposure time to risk rather than a risk factor. Estimates of mortality rates were based on observations in age categories.⁹

Pocock et al¹⁰ published a prediction instrument based on data from 8 randomized controlled trials of antihypertensive treatment in asymptomatic individuals: the INDIANA project. They developed 3 separate models for overall cardiovascular mortality, fatal coronary heart disease, and fatal stroke.

The validation cohort consisted of 592 consecutive patients included in the Rotterdam Transcranial Doppler study.¹⁶ Patients were recruited in our hospital. This study was designed to evaluate the diagnostic and prognostic value of transcranial Doppler ultrasonography in patients with a recent TIA or minor ischemic stroke. Patients were 18 years and older and had the index event within the preceding 6 months. They were independent (a score of ≤3 on the modified Rankin scale),¹⁷ and the TIA or stroke was of presumed atherosclerotic origin; this implied that patients with a mechanical heart valve and a proven dissection were excluded. Patients with atrial fibrillation or a significant symptomatic carotid artery stenosis were also excluded. Follow-up started at time of recruitment. The mean time between index event and recruitment was 51±45 days. The mean follow-up in this study was 755 days or 2.1 years (interquartile range, 0.9–3.1). The study population consisted of patients for whom no other treatment than risk factor modification and antiplatelet medication were available.

The primary outcome was the composite end point of stroke, myocardial infarction, and vascular death. The secondary outcome was the composite end point of fatal or nonfatal stroke. In the validation cohort, stroke was defined as a focal neurological deficit, resulting in disability for >24 hours, confirmed by a neurologist, with CT scan, if possible.

Myocardial infarction was defined as an episode of precordial chest pain accompanied by ECG evidence of recent infarction or release of cardiac enzymes. Vascular death was death within 4 weeks after stroke or myocardial infarction, or sudden death. Follow-up was performed annually by telephone survey by an experienced nurse. If patients reported any hospital visits or any symptoms suggesting a vascular event, then their treating physicians were contacted for further information or discharge letters. In the validation cohort, Kaplan-Meier analysis of survival was used to estimate the 2-year risk of the primary and secondary outcome.

The validation procedure was similar for each prediction model. First, we estimated the actual 2-year risk of primary and secondary outcome events for each patient in the validation cohort using Kaplan-Meier survival analysis. For missing values, the mean value of a variable was imputed. We then calculated the 2-year risk as predicted by the seven models. To do so, we used the original prediction equations of each model. Each equation was adjusted for the mean overall risk and for the prevalence of risk factors in the validation cohort, as recommended by several authors.^18,19

Two aspects of validity were examined: calibration and discrimination. Calibration, or reliability, measures how closely predicted outcomes agree with actual outcomes. Discrimination refers to the ability to distinguish patients with different outcomes.²⁰ The predicted probabilities of vascular events should be trustworthy (calibration) and extreme (discrimination).

Calibration-in-the-large reflects whether the overall outcome of the study cohort was close to the average predicted 2-year risk from a model. We constructed calibration plots in which observed 2-year outcome was plotted against predicted 2-year risk (using a Kaplan-Meier survival curve). We indicate outcome by quintiles of predicted probabilities. Calibration curves can be approximated by a regression line (or calibration line) with intercept (α) and slope (β). Well-calibrated models have α=0 and β=1. We calculated slopes for each prediction model. Discrimination was quantified with the concordance (c) statistic. The c-statistic resembles the area under the receiver-operating characteristic curve. A c-statistic of 1 implies a test with perfect sensitivity and specificity, whereas a value of 0.5 implies that the model predictions are no better than chance.²⁰

We first compared the distribution of prognostic factors in the 7 derivation cohorts (Table 2). People in the population-based studies from the INDIANA project and Framingham were younger than patients in the stroke cohorts and, by definition, did not have a history of stroke or myocardial infarction. For the SCORE population, the mean age or age distribution was not stated. In this study, relatively young subjects had participated, and subjects aged 40 to 60 were particularly well-represented. In the Oxford TIA cohort, all patients had a TIA, and approximately one-third had amaurosis fugax (Table 2). In the validation cohort, 57 major vascular events occurred during follow-up, for a 2-year risk of 12%, and 41 fatal or nonfatal stroke, for a 2-year-risk of 9% (Table 3).

Table 4 shows the observed predicted 2-year risk of stroke, myocardial infarction, or vascular death. Table 5 shows the observed predicted 2-year risk of fatal or nonfatal stroke. The SPI-II overestimated the risk for both outcome events in patients at low risk and underestimated in patients at high risk, whereas Oxford TIA, the Dutch TIA trial, and ABCD² score overestimated the risk for both outcome events (Figure).

Discrimination varied between 0.61 and 0.68 for the risk of stroke, myocardial infarction, or vascular death, and between 0.58 and 0.65 for the risk of fatal or nonfatal stroke (Table 4). Discrimination was higher in the stroke cohort-based studies than in the population-based studies but remained <0.70.

Some limitations of this study should be discussed. First, although the validation cohort was of reasonable size (598 patients), the effective sample size was not large, with 57 major events and 41 fatal or nonfatal strokes. Second, predictions from all models, except for the ABCD² study, exceeded the 2- year risk observed in our validation cohort. We have overcome this miscalibration by adjusting for this difference in follow-up in our analysis.

We excluded patients with atrial fibrillation or a severe carotid artery stenosis and focused on patients with TIA or minor ischemic stroke. This reduces the heterogeneity of the validation cohort and may reduce the discriminatory ability of the models. Risk estimates are more readily available for carotid artery stenosis, as well as for patients with atrial fibrillation. For the remaining group of patients with TIA or minor stroke, there is a particular need for a prediction model.

In the validation cohort as well as in 2 of the 4 stroke cohorts, the mean age was rather low (Table 2). These cohorts consisted of a hospital population. Hospital populations tend to be of lower age. Therefore, they may not be representative of patients with TIA and minor strokes in the community.

The time between index event and recruitment in the validation cohort was 51 days, which differs substantially from the ABCD² study but not the remaining stroke-based population studies. We have overcome this problem by adjusting for the baseline risk in all validation studies. In this way, we adjusted for the low number of outcome events compared to the ABCD² study.

Although this was a study on long-term risk, we also included the ABCD² model, which was developed for estimation of short-term risk. Calibration and discrimination in the ABCD² study, however, did not differ substantially from the long-term prediction models. Comparisons with other studies cannot be made, because similar studies of long-term risk assessment in patients with TIA or ischemic are not available.

The 7 models used dichotomized variables instead of continuous variables. Perhaps with continuous variables, estimations of risk would have been more precise and discrimination may improve.²² Furthermore, the stroke-based prediction models were derived from trial populations with certain inclusion and exclusion criteria. The best-fitting model for everyday practice will probably be derived from a representative cohort; this means a consecutive stroke population without exclusion criteria.