Validating the Stroke-Thrombolytic Predictive Instrument in a Population in the United Kingdom

Introduction

Predictive equations are useful to support clinical decision-making about thrombolysis with recombinant tissue plasminogen activator in acute stroke and to communicate risk/benefit information to patients and families.¹ The Stroke-Thrombolytic Predictive Instrument² (S-TPI) provides patient-specific predictions at 3 months for the likelihood of a normal/near-normal outcome (modified Rankin scale score, ≤1: no symptoms or slight disability), referred to as a normal outcome hereafter, and of catastrophic outcome (modified Rankin Scale score ≥5: severe disability/death).

A single-center cohort study (N=301) reported the S-TPI had reasonable external validity when applied to patients treated in routine practice but overestimated and underestimated probabilities for normal and catastrophic outcomes, respectively.³ We aimed to identify sources of prediction discrepancies between the S-TPI and outcomes in a larger population of patients treated in routine practice and to identify extensions that enhance the explanatory properties of the S-TPI.

Materials and Methods

Calibration curves were used to establish how predictions from the S-TPI corresponded with outcomes in the Safe Implementation of Treatments in Stroke United Kingdom (SITS-UK)⁴ population treated with recombinant tissue plasminogen activator between December 2002 and February 2010 in United Kingdom centers (N=4022).

Stepwise logistic regression was used to identify predictor variables associated with underprediction or overprediction of outcomes in SITS-UK patients. We also tested whether 3 additional patient characteristics (congestive heart failure, signs of current infarction on pretreatment brain scan, and serum glucose⁵) would improve the explanatory power of the model for normal outcomes in treated patients. For normal outcomes, a parsimonious method of calibration that estimated only an intercept and single calibration coefficient was rejected because of uncertainty about the differing relative strength of the predictors in the 2 datasets.⁶ Because the S-TPI assumes no association between treatment with recombinant tissue plasminogen activator and a catastrophic outcome, and because death before 3 months is a competing risk to a normal outcome at 3 months, only those surviving (modified Rankin Scale score, 0–5) at 3 months were used in the calibration of normal outcomes.

Receiver-operating curves were used to estimate the ability of the S-TPI to discriminate between those most and least likely to benefit from treatment, and between those most and least likely to experience a catastrophic outcome.

Results

The characteristics of the SITS-UK patients and of those used to develop the S-TPI are shown in Table 1. The calibration of normal outcomes included 1860 cases (1583 cases were excluded because the dependency state was not recorded; 352 had died within 3 months; 123 had treatment times or systolic blood pressures outside the defined ranges; 104 were missing systolic blood pressure, glucose, or stroke severity information). The calibration of a catastrophic outcome included 2212 cases.

Original S-TPI Predictions of Outcomes in SITS-UK Data

Calibration curves for predicted probability of the S-TPI of normal and catastrophic outcomes in the SITS-UK population are shown in FigureA and B. The S-TPI underpredicts the probability of catastrophic outcomes in the SITS-UK population; for example, a predicted P=0.60 equates to an actual observed P=0.50 (Figure A). The S-TPI overpredicts the probability of normal outcomes in the SITS-UK population (Figure B). At low probabilities of normal outcome, the overprediction is reversed and the S-TPI underpredicts.

• Download figure
• Download PowerPoint

Calibration for Normal Outcomes

The parameter estimates for the calibrated S-TPI are shown in Table 2. The S-TPI prediction is retained (1.3770; P=0.0117). No prediction discrepancy is associated with diabetes, previous stroke, and systolic blood pressure. Prediction discrepancy is associated with male gender, age, and National Institutes of Stroke Scale score. Of the additional predictors, infarction on pretreatment brain scan and serum glucose are also found to be associated with a normal outcome. Figure C shows the improved areas under the curve (0.754–0.766) for the calibrated S-TPI models for all cases, including those who did not survive to 3 months, reflecting the S-TPI finding of an absence of association between treatment with recombinant tissue plasminogen activator and death.

Calibration for Catastrophic Outcomes

The SITS-UK population risk of catastrophic outcome was greater than predicted by the S-TPI (Table 2). No receiver-operating curve is shown for catastrophic outcome because the parsimonious recalibrating does not affect the ranking of case, but the area under the curve is 0.784.

Discussion

Consistent with previous research, we found evidence that the S-TPI overestimates the probability of a normal outcome and underestimates the probability of a catastrophic outcome in treated patients. The strength of the calibrated S-TPI model is its applicability to current practice because the predictions are adjusted using data about patients routinely treated up to year 2010, and it includes additional patient characteristics.

In terms of weaknesses, there may have been bias in the routine practice data. For example, 1 possible reason for the overprediction of normal outcomes is that United Kingdom clinicians (compared with European/North America clinicians) may assign lower modified Rankin Scale scores to patients with similar levels of disability. Studies assessing inter-rater reliability of modified Rankin assessments show only modest agreement, with a kappa of <0.5.⁷ Prediction -discrepancies associated with men and additional predictors mean that untreated outcomes cannot be estimated using the calibrated model. Like the S-TPI, our model predicts no overall harm from treatment; its use as a guide for clinical decision-making is only warranted when thrombolytic treatment is considered to have no association with increased mortality (an assumption more valid at a population level than an individual level) or used with separate predictors of harmful outcomes.

Conclusion

Notwithstanding the assumption about the association between treatment and death, our findings suggest that recalibrated S-TPI is a good basis for predicting outcomes at 3 months in treated patients and its explanatory power can be improved to reflect outcomes seen in routine practice.

Acknowledgments

The authors express thanks to Kennedy R. Lees (Professor of Cerebrovascular Medicine, University of Glasgow) for access to the -SITS-UK data. We thank all UK sites that contributed data to the SITS-UK database.

Footnotes