Prediction of Recurrent Ischemic Stroke Using Registry Data and Machine Learning Methods: The Erlangen Stroke Registry

The best prediction performance was achieved by the linear support vector machine algorithm¹⁹ in the combination with Synthetic Minority Oversampling Technique. The AUROC was 0.70 (95% CI, 0.64–0.76), as measured by double nested cross-validation (Figure 2A). The pooled confusion matrix over 5 test cross-validation folds showed specificity of 0.78 and sensitivity of 0.63 (Figure 2B). These 2 metrics are selected as they represent common metrics for evaluating the utility of binary classification models in medical applications. An additional evaluation on the hold-out set showed comparable results (AUROC of 0.72). Performances of other approaches were significantly lower (Table 2). The learning curve reveals steadily growing AUROC when more data are provided to the learning algorithm (Figure 2C). The training AUROC depicted by the upper line finally comes close to the test AUROC, indicating that the model does not suffer from significant overfitting. The learning curve also shows that model test performance is likely to grow further with more follow-ups but likely not above the upper bound defined by the training AUROC (0.74). The top 10 variables influencing the predictions, according to the SHAP framework, were widow(er) marital status, received rehabilitation, living situation, age, high calorie food, mean diastolic blood pressure, percutaneous endoscopic gastrotomy, GP home visits, Mini Mental Status Test, and mean systolic blood pressure (Figure 2D). The direction of influence of these variables on the risk of recurrent stroke is illustrated in Figure S3 and discussed in the next section. See Table S2 for list of all 20 variables in the final model. Model calibration showed significant improvement after applying isotonic regression compared to uncalibrated support vector machine model (Figure 2E and 2F). Nonetheless, the calibrated model still overestimates of the risk of recurrent stroke considerably, especially in the mid and high ranges of the stroke probability.

In this study, we developed a fairly accurate machine learning model for estimating the individual risk of recurrent ischemic stroke within 1 year solely based on easily obtained patient data. To the best of our knowledge, this model is the first to reach objectively measured AUROC of 0.70 without depending on imaging data. No comprehensive clinical data were available—very limited medication information and no laboratory values or images were available. Information on all 20 variables necessary for the risk prediction can be collected via interviews with patients in the home care setting. The dynamic nature of our model makes it possible to recalculate the individual risk whenever new data become available.

To compare our approach with other prediction instruments targeting the same long-term prediction, we reviewed published statistical and machine learning studies and the established clinical risk scores. A meta-analysis estimated an AUROC range of 0.55 to 0.65 and 0.58 to 0.68 for ESRS and modified ESRS, respectively,⁷ which are the scores developed for the time horizon of 1 year. Our model showed better performance, however requiring 20 variables for computing prediction versus 8 and 11 of ESRS and modified ESRS, respectively. At the first glance, our best model might seem suboptimal compared with the published performance of the logistic regression (AUROC 0.74)¹⁵ and the artificial neural network (AUROC 0.77).¹⁷ The performance of both of those approaches, however, was not well established as it was measured on the same data used for training in the logistic regression approach and bias was introduced by manually balancing classes in the data before modeling and evaluation in the artificial neural network approach. Moreover, both approaches require more comprehensive features, which cannot be easily obtained from the patient at home. The logistic regression approach is dependent on both the clinical values and retinal imaging features. The artificial neural network approach requires demographic, clinical and medication data as well as features extracted manually from Doppler, CT, MR, or digital subtraction angiography.

We tried to avoid any algorithm selection bias and personal preferences by evaluating a comprehensive set of 25 different algorithms. Where applicable, we combined them with 3 techniques for treating the class imbalance problem, which resulted in 52 machine learning approaches tested. This is one of the major strengths of this study. The prediction performance obtained by the rigorous double-nested cross-validation method was additionally confirmed on the separate hold-out test set of patients. It is important to note that our validation method did not only validate the prediction model itself but also all other modeling steps included in the machine learning pipeline (Figure 1). By making sure that all follow-ups from a single patient are either in the training, or the test set of each double nested cross-validation fold, we prevented potential data leakage and made the performance estimation additionally conservative.

The learning curve for the best model confirmed that no serious overfitting took place. Moreover, it showed that the performance of the linear support vector machine model would likely improve slightly if more data were provided to the learning algorithm. After performance evaluation, the final prediction model was trained on all available data which was 28% larger than the original training sets. Therefore, it is reasonable to expect even slightly better performance on further unseen, future data.

By applying isotonic regression, the calibration curve of the prediction model was significantly improved. Nevertheless, it indicated a considerable deficit; namely, that the model overestimates the risk in mid and high probability range. However, from a clinical point of view, the risk overestimation is preferred to risk underestimation because false negatives are more critical than false positives.

The SHAP explainable AI framework revealed which variables were most important for the risk assessment. Although this is solely a model-specific indicator of the relative variable importance as measured in the hold-out test, it is worth noting that several widely considered stroke and recurrent stroke risk factors showed up among the top 10 predictors, such as patient age, systolic and diastolic blood pressure. The usefulness of these variables in recurrent risk stroke prediction was also validated in several clinical scores as well, such as the ESRS and the California risk scores. This gives confidence that the predictions computed by the prediction model are to a large extent based on known clinically relevant variables. Another informative plot of the SHAP framework is the summary plot, which shows estimated feature effects on the model output (Figure S3). The effects of several variables are intuitive, for example, the lack of poststroke rehabilitative measures, higher age, higher mean diastolic blood pressure, and the need for more General Practitioner’s (GP) visits are all recognized as risk-increasing factors. Several effects might seem counterintuitive at first: patient widowed status, percutaneous endoscopic gastrotomy (PEG), living in a house, and high mean systolic blood pressure are estimated as risk-decreasing factors. However, these could be interpreted as follows: widowed and patients with PEG might be getting more support from their surrounding (eg, by a family, friends, or a care nurse)—a potential confounding factor, while living in a house might be a sign of a higher socioeconomic status, which may correlate with better physical health in general. High mean systolic blood pressure as a risk-decreasing factor might be an artifact of certain drugs. For example, it is not uncommon that NSAIDS (potential further confounder) used for the prevention of recurrent stroke raise blood pressure. Other estimated risk-decreasing factors including high calorie diet and lower scores of the Mini Mental State Test are not easy to explain and might be an artifact of the imperfect model. The SHAP summary plot describes the behavior of the (imperfect) predictive model and not necessarily the causal relationships between variables.

Several important limitations of this study must be underlined. All data came from the same source (ESPro registry), and the algorithm was not validated externally. Despite significant calibration improvement using isotonic regression, the final calibration curve was still suboptimal due to risk overestimation. Variable importance and effects estimation was based on one method only, while ideally several methods could have been applied and compared. These technical points remain the subject of potential future work. Moreover, the number of available follow-ups was relatively small and the number of initially collected variables high (250). To focus on those variables, which are potentially clinically relevant, 2 stroke experts made a preselection of variables to be included in the analysis. In this process, a personal bias could not be excluded. While 4 of the top 5 binary variables in the final model (Table S2) are also selected in all 5 models of the outer double-nested cross-validation loop (Table S3), there is still a significant variability in χ² variable selection. This can be attributed to the relatively small sample size. Finally, further bias might have been created by the inclusion of only expired patients in the analysis, which was one of the data privacy requirements in this study.

Our modeling study showed that a reasonably accurate individual prediction of recurrent stroke within 1 year from the patient interview is feasible using ESPro registry data. The developed model is dynamic and applicable at any time point when the necessary patient data become available, and not just at the timepoint of stroke onset. While related work indicates that possibly more accurate models could be developed using laboratory and imaging data, our prediction model can be used for regular and more frequent initial risk assessments in the patient’s home care setting, potentially as a part of web-based or mobile telehealth solution. Identified high-risk patients could be monitored more closely and potentially advised to consult their treating physician about necessary therapy adjustments. Such software tools could also be beneficial for providers who monitor their patients via structured telemonitoring programs. Risk prediction could enable patient risk stratification, empowering providers to focus on high-risk patients.