Machine Learning–Based Model for Prediction of Outcomes in Acute Stroke

The prediction of long-term outcomes in ischemic stroke patients may be useful in treatment decisions, as well as in managing prognostic expectations. Several prognostic scoring systems have been developed for this purpose.¹ In light of recent advances in machine learning, application of the technique in the medical field has yielded promising results.² The complex and unpredictable nature of human physiology has, in many circumstances, proven to be better described by the machine learning algorithms. Unlike the traditional predictive models that use selected variables for calculation, machine learning techniques can easily incorporate a large number of variables, as all calculations are performed using a computer.³ These characteristics make machine learning techniques suitable for the medical field. In stroke, machine learning techniques are increasingly used in various areas including outcome prediction after endovascular treatment.^4,5

With consideration of its expected impact on ischemic stroke management, we developed models using machine learning techniques to predict long-term stroke outcomes. We then compared the predictability to the Acute Stroke Registry and Analysis of Lausanne (ASTRAL) score, which is a well-known prognostic model.¹

Methods

The data that support the findings of this study are available from the corresponding author upon reasonable request. This was a retrospective study using a prospective cohort that registers patients with ischemic stroke who are admitted within 7 days of the onset of symptoms. For this study, we included all patients admitted between January 2010 and December 2014. We excluded patients with prestroke modified Rankin Scale (mRS) score of >2, missing mRS score at 3 months, or those who received recanalization treatment. Functional outcome was determined at 3 months (online-only Data Supplement), and favorable outcome was defined as mRS score 0, 1, or 2. This study was approved by the Institutional Review Board of Yonsei University Health System with a waiver of informed consent due to the retrospective nature of the study.

Data and Machine Learning Algorithms

For the development of machine learning models, we obtained 38 variables including patient demographics, initial National Institutes of Health Stroke Scale scores, time from onset to admission, stroke subtypes based on the Trial of ORG 10472 in Acute Stroke Treatment classification system, history of previous diseases and medications, laboratory findings, and mRS scores at 3 months (Table I in the online-only Data Supplement).

We used 3 machine learning algorithms: the deep neural network, random forest, and logistic regression.^3,6 The deep neural network comprises layers of interconnected artificial neurons. An artificial neuron is designed based on the biological neuron itself and receives multiple inputs multiplied by weights and outputs the sum of the inputs. The random forest algorithm consists of a multitude of decision trees comprising multiple true or false conditions using input variables. The sum of the decisions made by the decision trees is used for the final classification.

Machine learning models were trained with all variables as inputs to classify patients likely to have favorable outcomes. For the deep neural network model, 3 hidden layers with 15 artificial neural network units each were used. For the random forest model, 300 decision trees were used. To evaluate the accuracy of the machine learning models, we calculated the ASTRAL score as a reference, which is one of the established prognostic scoring systems for acute stroke.

We also investigated how machine learning models predict outcomes when the 6 variables for calculating the ASTRAL score were used as inputs: age, National Institute of Health Stroke Scale score, onset to admission delay, visual field defect, glucose, and decreased level of consciousness. For this analysis, the machine learning models were trained with 6 variables from the ASTRAL score. One hidden layer with 4 artificial neural network units was used for the deep neural network model, and 150 decision trees were used for the random forest model.

Among the study population, 67% (n=1744) were randomly selected for the training set and the remaining 33% (n=858) were used as the test set to prevent overfitting of the models. TensorFlow version 1.1.0 (Google) and scikit-learn toolkit version 0.18.1 (Google) were used to train the machine learning models.⁷

Statistical Analyses

Statistical analyses were performed using R package version 3.3.2. Receiver operating characteristic curve analysis and the area under the curve were calculated using the pROC package to compare the efficacy of each model. Variables with P of <0.05 were considered statistically significant, and all P were 2-sided.

Results

A total of 3522 patients were registered to the cohort during the study period. After excluding 453 patients with unavailable 3-month mRS scores, 60 patients with prestroke mRS scores of >2, 87 patients with missing laboratory tests or clinical data, and 318 patients who underwent thrombolytic therapy, 2604 patients were finally included (Figure 1). The mean age of the 2604 patients was 66.2±12.6 years and 61.7% were men. Comparison of demographic variables between the enrolled and excluded patients are shown in the online-only Data Supplement (Table II in the online-only Data Supplement).

• Download figure
• Download PowerPoint

Comparison of the Models for the Prediction of Favorable Outcomes

Of the 2604 patients, 2043 (78%) patients had favorable outcomes. The deep neural network model performed significantly better than the ASTRAL score (area under the curve 0.888 [95% CI, 0.873–0.903] versus 0.839 [0.822–0.855]; P<0.001). However, the performance of the random forest model (0.857 [0.840–0.874]) and the logistic regression model (0.849 [0.831–0.867]) did not differ from that of the ASTRAL score (P=0.136, P=0.413, respectively; Figure 2).

• Download figure
• Download PowerPoint

When we used only the variables from the ASTRAL score, the performance of the machine learning models did not differ from that of ASTRAL score (0.853 [95% CI, 0.835–0.871] for the deep neural network model, P=0.255; 0.828 [95% CI, 0.808–0.847] for the random forest model, P=0.396; and 0.846 [95% CI, 0.828–0.865] for the logistic regression model, P=0.541; Table III in the online-only Data Supplement).

Discussion

This study demonstrated that the use of machine learning models can accurately predict long-term outcomes in acute stroke patients. Prognostic scores calculated based on statistical analyses, for their simplicity, use only a few crucial variables, with their coefficients roughly rounded. However, many factors influence stroke outcomes, and these variables may have, even a slight, impact on prediction. Indeed, our study demonstrated that the deep neural network model performed significantly better with the incorporation of less crucial variables than the ASTRAL score while performed similarly to the ASTRAL score when only the variables from the ASTRAL score were used as inputs.

This study demonstrated that the deep neural network model performed better than the other models. The deep neural network model itself may be more suitable for the prediction of outcomes. Multiple layers of complex network may be effective in representing the complex nature of the outcomes in a stroke patient. However, the theoretical background underlying the improved performance is not known.

Improvement of predictability from the ASTRAL score was small, especially considering the added burden of entering many variables for the machine learning model. However, with the improvement of electronic health record systems, automatic calculations are built into the system, thereby diminishing the need for the models to be simple.⁸ In addition, considering that machine learning models can be self-taught with additional data, the aforementioned results are subject to improvement but promising.

There are several limitations in this study. This was a single-center study and requires validation with data from other sources. Variables used as inputs to the machine learning algorithms were those that are typically obtainable or evaluated in most cases. However, the prediction might be influenced slightly according to the variables and may be adjusted with consideration for their availability when incorporating data from other centers. Patients with recanalization therapy were excluded because the prognosis of those patients is majorly influenced by variables from the treatment itself, and these variables are only applicable for the subgroup of patients who received the treatment.

Conclusions

This study demonstrated that machine learning algorithms, particularly the deep neural network, can improve long-term outcome prediction for ischemic stroke patients.

Footnotes