Machine Learning–Based Model for Prediction of Outcomes in Acute Stroke
Background and Purpose—
The prediction of long-term outcomes in ischemic stroke patients may be useful in treatment decisions. Machine learning techniques are being increasingly adapted for use in the medical field because of their high accuracy. This study investigated the applicability of machine learning techniques to predict long-term outcomes in ischemic stroke patients.
This was a retrospective study using a prospective cohort that enrolled patients with acute ischemic stroke. Favorable outcome was defined as modified Rankin Scale score 0, 1, or 2 at 3 months. We developed 3 machine learning models (deep neural network, random forest, and logistic regression) and compared their predictability. To evaluate the accuracy of the machine learning models, we also compared them to the Acute Stroke Registry and Analysis of Lausanne (ASTRAL) score.
A total of 2604 patients were included in this study, and 2043 (78%) of them had favorable outcomes. The area under the curve for the deep neural network model was significantly higher than that of the ASTRAL score (0.888 versus 0.839; P<0.001), while the areas under the curves of the random forest (0.857; P=0.136) and logistic regression (0.849; P=0.413) models were not significantly higher than that of the ASTRAL score. Using only the 6 variables that are used for the ASTRAL score, the performance of the machine learning models did not significantly differ from that of the ASTRAL score.
Machine learning algorithms, particularly the deep neural network, can improve the prediction of long-term outcomes in ischemic stroke patients.
The prediction of long-term outcomes in ischemic stroke patients may be useful in treatment decisions, as well as in managing prognostic expectations. Several prognostic scoring systems have been developed for this purpose.1 In light of recent advances in machine learning, application of the technique in the medical field has yielded promising results.2 The complex and unpredictable nature of human physiology has, in many circumstances, proven to be better described by the machine learning algorithms. Unlike the traditional predictive models that use selected variables for calculation, machine learning techniques can easily incorporate a large number of variables, as all calculations are performed using a computer.3 These characteristics make machine learning techniques suitable for the medical field. In stroke, machine learning techniques are increasingly used in various areas including outcome prediction after endovascular treatment.4,5
With consideration of its expected impact on ischemic stroke management, we developed models using machine learning techniques to predict long-term stroke outcomes. We then compared the predictability to the Acute Stroke Registry and Analysis of Lausanne (ASTRAL) score, which is a well-known prognostic model.1
The data that support the findings of this study are available from the corresponding author upon reasonable request. This was a retrospective study using a prospective cohort that registers patients with ischemic stroke who are admitted within 7 days of the onset of symptoms. For this study, we included all patients admitted between January 2010 and December 2014. We excluded patients with prestroke modified Rankin Scale (mRS) score of >2, missing mRS score at 3 months, or those who received recanalization treatment. Functional outcome was determined at 3 months (online-only Data Supplement), and favorable outcome was defined as mRS score 0, 1, or 2. This study was approved by the Institutional Review Board of Yonsei University Health System with a waiver of informed consent due to the retrospective nature of the study.
Data and Machine Learning Algorithms
For the development of machine learning models, we obtained 38 variables including patient demographics, initial National Institutes of Health Stroke Scale scores, time from onset to admission, stroke subtypes based on the Trial of ORG 10472 in Acute Stroke Treatment classification system, history of previous diseases and medications, laboratory findings, and mRS scores at 3 months (Table I in the online-only Data Supplement).
We used 3 machine learning algorithms: the deep neural network, random forest, and logistic regression.3,6 The deep neural network comprises layers of interconnected artificial neurons. An artificial neuron is designed based on the biological neuron itself and receives multiple inputs multiplied by weights and outputs the sum of the inputs. The random forest algorithm consists of a multitude of decision trees comprising multiple true or false conditions using input variables. The sum of the decisions made by the decision trees is used for the final classification.
Machine learning models were trained with all variables as inputs to classify patients likely to have favorable outcomes. For the deep neural network model, 3 hidden layers with 15 artificial neural network units each were used. For the random forest model, 300 decision trees were used. To evaluate the accuracy of the machine learning models, we calculated the ASTRAL score as a reference, which is one of the established prognostic scoring systems for acute stroke.
We also investigated how machine learning models predict outcomes when the 6 variables for calculating the ASTRAL score were used as inputs: age, National Institute of Health Stroke Scale score, onset to admission delay, visual field defect, glucose, and decreased level of consciousness. For this analysis, the machine learning models were trained with 6 variables from the ASTRAL score. One hidden layer with 4 artificial neural network units was used for the deep neural network model, and 150 decision trees were used for the random forest model.
Among the study population, 67% (n=1744) were randomly selected for the training set and the remaining 33% (n=858) were used as the test set to prevent overfitting of the models. TensorFlow version 1.1.0 (Google) and scikit-learn toolkit version 0.18.1 (Google) were used to train the machine learning models.7
Statistical analyses were performed using R package version 3.3.2. Receiver operating characteristic curve analysis and the area under the curve were calculated using the pROC package to compare the efficacy of each model. Variables with P of <0.05 were considered statistically significant, and all P were 2-sided.
A total of 3522 patients were registered to the cohort during the study period. After excluding 453 patients with unavailable 3-month mRS scores, 60 patients with prestroke mRS scores of >2, 87 patients with missing laboratory tests or clinical data, and 318 patients who underwent thrombolytic therapy, 2604 patients were finally included (Figure 1). The mean age of the 2604 patients was 66.2±12.6 years and 61.7% were men. Comparison of demographic variables between the enrolled and excluded patients are shown in the online-only Data Supplement (Table II in the online-only Data Supplement).
Comparison of the Models for the Prediction of Favorable Outcomes
Of the 2604 patients, 2043 (78%) patients had favorable outcomes. The deep neural network model performed significantly better than the ASTRAL score (area under the curve 0.888 [95% CI, 0.873–0.903] versus 0.839 [0.822–0.855]; P<0.001). However, the performance of the random forest model (0.857 [0.840–0.874]) and the logistic regression model (0.849 [0.831–0.867]) did not differ from that of the ASTRAL score (P=0.136, P=0.413, respectively; Figure 2).
When we used only the variables from the ASTRAL score, the performance of the machine learning models did not differ from that of ASTRAL score (0.853 [95% CI, 0.835–0.871] for the deep neural network model, P=0.255; 0.828 [95% CI, 0.808–0.847] for the random forest model, P=0.396; and 0.846 [95% CI, 0.828–0.865] for the logistic regression model, P=0.541; Table III in the online-only Data Supplement).
This study demonstrated that the use of machine learning models can accurately predict long-term outcomes in acute stroke patients. Prognostic scores calculated based on statistical analyses, for their simplicity, use only a few crucial variables, with their coefficients roughly rounded. However, many factors influence stroke outcomes, and these variables may have, even a slight, impact on prediction. Indeed, our study demonstrated that the deep neural network model performed significantly better with the incorporation of less crucial variables than the ASTRAL score while performed similarly to the ASTRAL score when only the variables from the ASTRAL score were used as inputs.
This study demonstrated that the deep neural network model performed better than the other models. The deep neural network model itself may be more suitable for the prediction of outcomes. Multiple layers of complex network may be effective in representing the complex nature of the outcomes in a stroke patient. However, the theoretical background underlying the improved performance is not known.
Improvement of predictability from the ASTRAL score was small, especially considering the added burden of entering many variables for the machine learning model. However, with the improvement of electronic health record systems, automatic calculations are built into the system, thereby diminishing the need for the models to be simple.8 In addition, considering that machine learning models can be self-taught with additional data, the aforementioned results are subject to improvement but promising.
There are several limitations in this study. This was a single-center study and requires validation with data from other sources. Variables used as inputs to the machine learning algorithms were those that are typically obtainable or evaluated in most cases. However, the prediction might be influenced slightly according to the variables and may be adjusted with consideration for their availability when incorporating data from other centers. Patients with recanalization therapy were excluded because the prognosis of those patients is majorly influenced by variables from the treatment itself, and these variables are only applicable for the subgroup of patients who received the treatment.
This study demonstrated that machine learning algorithms, particularly the deep neural network, can improve long-term outcome prediction for ischemic stroke patients.
Sources of Funding
This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2018R1A2A3074996).
Ntaios G, Faouzi M, Ferrari J, Lang W, Vemmos K, Michel P. An integer-based score to predict functional outcome in acute ischemic stroke: the ASTRAL score.Neurology. 2012; 78:1916–1922. doi: 10.1212/WNL.0b013e318259e221CrossrefMedlineGoogle Scholar
Asadi H, Dowling R, Yan B, Mitchell P. Machine learning for outcome prediction of acute ischemic stroke post intra-arterial therapy.PLoS One. 2014; 9:e88225. doi: 10.1371/journal.pone.0088225CrossrefGoogle Scholar
LeCun Y, Bengio Y, Hinton G. Deep learning.Nature. 2015; 521:436–444. doi: 10.1038/nature14539CrossrefMedlineGoogle Scholar
van Os HJA, Ramos LA, Hilbert A, van Leeuwen M, van Walderveen MAA, Kruyt ND,. Predicting outcome of endovascular treatment for acute ischemic stroke: potential value of machine learning algorithms.Front Neurol. 2018; 9:784.CrossrefMedlineGoogle Scholar
Cuadrado-Godia E, Dwivedi P, Sharma S, Ois Santiago A, Roquer Gonzalez J, Balcells M,. Cerebral small vessel disease: a review focusing on pathophysiology, biomarkers, and machine learning strategies.J Stroke. 2018; 20:302–320. doi: 10.5853/jos.2017.02922CrossrefMedlineGoogle Scholar
Breiman L. Random forests.Machine Learning. 2001; 45:5–32.CrossrefGoogle Scholar
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J,. Tensorflow: a system for large-scale machine learning.OSDI. 2016; 16:265–283.Google Scholar
Flint AC, Rao VA, Chan SL, Cullen SP, Faigeles BS, Smith WS,; SITS International and VISTA-plus Investigators. Improved ischemic stroke outcome prediction using model estimation of outcome probability: the THRIVE-c calculation.Int J Stroke. 2015; 10:815–821. doi: 10.1111/ijs.12529CrossrefGoogle Scholar