Validating the Stroke-Thrombolytic Predictive Instrument in a Population in the United Kingdom
Background and Purpose—
This study aimed to test the explanatory qualities of the Stroke-Thrombolytic Predictive Instrument (S-TPI) when applied to patients treated in routine practice.
S-TPI predictions were compared with observed outcomes in terms of normal/near-normal (modified Rankin Scale score, ≤1) and catastrophic outcome (modified Rankin Scale score, ≥5) at 3 months. Logistic regression was used to calibrate and expand the S-TPI.
The S-TPI overestimated probability of catastrophic outcomes and overestimated the probability of a normal/near normal outcome above 0.4 and underestimated those below. Calibrating the S-TPI minimized discrepancies between predicted and observed outcomes, in the case of normal/near-normal outcomes, where including additional predictors (serum glucose and signs of current infarction on pretreatment brain scan) further reduced discrepancies between predicted and observed outcomes.
The explanatory power of the S-TPI in thrombolytic-treated patients can be improved to reflect outcomes seen in routine practice.
Predictive equations are useful to support clinical decision-making about thrombolysis with recombinant tissue plasminogen activator in acute stroke and to communicate risk/benefit information to patients and families.1 The Stroke-Thrombolytic Predictive Instrument2 (S-TPI) provides patient-specific predictions at 3 months for the likelihood of a normal/near-normal outcome (modified Rankin scale score, ≤1: no symptoms or slight disability), referred to as a normal outcome hereafter, and of catastrophic outcome (modified Rankin Scale score ≥5: severe disability/death).
A single-center cohort study (N=301) reported the S-TPI had reasonable external validity when applied to patients treated in routine practice but overestimated and underestimated probabilities for normal and catastrophic outcomes, respectively.3 We aimed to identify sources of prediction discrepancies between the S-TPI and outcomes in a larger population of patients treated in routine practice and to identify extensions that enhance the explanatory properties of the S-TPI.
Materials and Methods
Calibration curves were used to establish how predictions from the S-TPI corresponded with outcomes in the Safe Implementation of Treatments in Stroke United Kingdom (SITS-UK)4 population treated with recombinant tissue plasminogen activator between December 2002 and February 2010 in United Kingdom centers (N=4022).
Stepwise logistic regression was used to identify predictor variables associated with underprediction or overprediction of outcomes in SITS-UK patients. We also tested whether 3 additional patient characteristics (congestive heart failure, signs of current infarction on pretreatment brain scan, and serum glucose5) would improve the explanatory power of the model for normal outcomes in treated patients. For normal outcomes, a parsimonious method of calibration that estimated only an intercept and single calibration coefficient was rejected because of uncertainty about the differing relative strength of the predictors in the 2 datasets.6 Because the S-TPI assumes no association between treatment with recombinant tissue plasminogen activator and a catastrophic outcome, and because death before 3 months is a competing risk to a normal outcome at 3 months, only those surviving (modified Rankin Scale score, 0–5) at 3 months were used in the calibration of normal outcomes.
Receiver-operating curves were used to estimate the ability of the S-TPI to discriminate between those most and least likely to benefit from treatment, and between those most and least likely to experience a catastrophic outcome.
The characteristics of the SITS-UK patients and of those used to develop the S-TPI are shown in Table 1. The calibration of normal outcomes included 1860 cases (1583 cases were excluded because the dependency state was not recorded; 352 had died within 3 months; 123 had treatment times or systolic blood pressures outside the defined ranges; 104 were missing systolic blood pressure, glucose, or stroke severity information). The calibration of a catastrophic outcome included 2212 cases.
|Characteristic||S-TPI (N=2131)||Omitted Cases Outcome Recorded SITS-UK (n=227)||SITS-UK Patients Surviving at 3 Months (n=1860)||SITS-UK Patients Not Surviving to 3 Months (n=352)|
|Age, mean (SD)||65.9 (11.4)||67.8 (13.3)||66.3 (12.8)||72.8 (11.6)|
|Gender, % male||54.7%||58.6%||59.7%||57.7%|
|NIHSS score (median, IQR)||12 (8, 17)||12 (7, 18)*||12 (8, 17)||18 (14, 22)|
|Previous stroke, %||16.6||14.41||11.9||14.2|
|Atrial fibrillation, %||18.6||27.0||23.0||27.6|
|OTT, min (median, IQR)||235 (155, 290)||146 (109, 175)||150 (120, 175)||151.5 (120, 180)|
|OTT, % within 3 to 4.5 hours||61.3||15.3||16.5||18.8|
|Systolic blood pressure, mm Hg (mean, SD)||152.6 (20.3)||145.5 (21.8)*||147.0 (20.9)||148.9 (21.0)|
|Serum glucose mmol/L (median, IQR)||6.8 (5.8, 8.6)||6.2 (5.6, 7.8)||6.2 (5.4, 7.5)||6.9 (5.9, 8.4)|
|Signs of current infarction on pretreatment scan, %||NA||23.9||28.8||35.5|
|Congestive heart failure, %||12.1||4.5||4.0||4.5|
Original S-TPI Predictions of Outcomes in SITS-UK Data
Calibration curves for predicted probability of the S-TPI of normal and catastrophic outcomes in the SITS-UK population are shown in FigureA and B. The S-TPI underpredicts the probability of catastrophic outcomes in the SITS-UK population; for example, a predicted P=0.60 equates to an actual observed P=0.50 (Figure A). The S-TPI overpredicts the probability of normal outcomes in the SITS-UK population (Figure B). At low probabilities of normal outcome, the overprediction is reversed and the S-TPI underpredicts.
Calibration for Normal Outcomes
The parameter estimates for the calibrated S-TPI are shown in Table 2. The S-TPI prediction is retained (1.3770; P=0.0117). No prediction discrepancy is associated with diabetes, previous stroke, and systolic blood pressure. Prediction discrepancy is associated with male gender, age, and National Institutes of Stroke Scale score. Of the additional predictors, infarction on pretreatment brain scan and serum glucose are also found to be associated with a normal outcome. Figure C shows the improved areas under the curve (0.754–0.766) for the calibrated S-TPI models for all cases, including those who did not survive to 3 months, reflecting the S-TPI finding of an absence of association between treatment with recombinant tissue plasminogen activator and death.
|Calibrating S-TPI Using SITS-UK Patients Surviving at 3 Months|
|Age per 1-y increase||0.0173||−0.0202||0.0412|
|Male vs female||−0.0529||−0.168||0.1078|
|Diabetes||−0.7431||. . .||. . .|
|NIHSS unit increase||−0.0076||−0.1576||0.0012|
|Previous stroke||−0.4010||. . .||. . .|
|Onset time to treatment per 1-min increase||−0.0034||. . .||. . .|
|Serum glucose per unit mmol/L increase||. . .||−0.0506||0.0360|
|Signs of current infarction on pretreatment brain scan||. . .||−0.4630||0.0001|
|Age *NIHSS score||−0.0029||0.0011||0.15773|
|Systolic blood pressure (per 1-mm Hg increase)||−0.0166||. . .||. . .|
|All SITS-UK patients (n=2, 212)|
Calibration for Catastrophic Outcomes
The SITS-UK population risk of catastrophic outcome was greater than predicted by the S-TPI (Table 2). No receiver-operating curve is shown for catastrophic outcome because the parsimonious recalibrating does not affect the ranking of case, but the area under the curve is 0.784.
Consistent with previous research, we found evidence that the S-TPI overestimates the probability of a normal outcome and underestimates the probability of a catastrophic outcome in treated patients. The strength of the calibrated S-TPI model is its applicability to current practice because the predictions are adjusted using data about patients routinely treated up to year 2010, and it includes additional patient characteristics.
In terms of weaknesses, there may have been bias in the routine practice data. For example, 1 possible reason for the overprediction of normal outcomes is that United Kingdom clinicians (compared with European/North America clinicians) may assign lower modified Rankin Scale scores to patients with similar levels of disability. Studies assessing inter-rater reliability of modified Rankin assessments show only modest agreement, with a kappa of <0.5.7 Prediction -discrepancies associated with men and additional predictors mean that untreated outcomes cannot be estimated using the calibrated model. Like the S-TPI, our model predicts no overall harm from treatment; its use as a guide for clinical decision-making is only warranted when thrombolytic treatment is considered to have no association with increased mortality (an assumption more valid at a population level than an individual level) or used with separate predictors of harmful outcomes.
Notwithstanding the assumption about the association between treatment and death, our findings suggest that recalibrated S-TPI is a good basis for predicting outcomes at 3 months in treated patients and its explanatory power can be improved to reflect outcomes seen in routine practice.
The authors express thanks to Kennedy R. Lees (
Sources of Funding
This article presents independent research commissioned by the
Dr Ford’s institution has received research grants from Boehringer Ingelheim (manufacturer of Alteplase) and honoraria from Lundbeck for stroke-related activities. Dr Ford also has received personal -remuneration for educational and advisory work from Boehringer Ingelheim and Lundbeck.
Flynn D, Murtagh M, Stobbart L, Rodgers H, Ford GA, Thomson RG. A review of decision support interventions for thrombolysis in acute stroke.Cerebrovasc Dis.2010; 29(Suppl 2):208.Google Scholar
Kent DM, Selker HP, Ruthazer R, Bluhmki E, Hacke W. The stroke–thrombolytic predictive instrument: a predictive instrument for intravenous thrombolysis in acute ischemic stroke.Stroke. 2006; 37:2957–2962.LinkGoogle Scholar
Uyttenboogaart M, Stewart RE, Vroomen PC, Luijckx GJ, De Keyser J. Utility of the stroke-thrombolytic predictive instrument.J Neurol Neurosurg Psychiatry. 2008; 79:1079–1081.CrossrefMedlineGoogle Scholar
- 4. SITS-ISTR. Safe Implementation of Treatments in Stroke - International Stroke Thrombolysis Registry.Available at:https://sitsinternational.org/. Accessed March 4, 2011.Google Scholar
Wahlgren N, Ahmed N, Eriksson N, Aichner F, Bluhmki E, Dávalos A,. Multivariable analysis of outcome predictors and adjustment of main outcome results to baseline data profile in randomized controlled trials: Safe Implementation of Thrombolysis in Stroke-MOnitoring STudy (SITS-MOST). Stroke. 2008; 39:3316–3322.LinkGoogle Scholar
Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage.Stat Med. 2004; 23:2567–2586.CrossrefMedlineGoogle Scholar
Quinn TJ, Dawson J, Walters MR, Lees KR. Reliability of the modified Rankin Scale: a systematic review.Stroke. 2009; 40:3393–3395.LinkGoogle Scholar