Assessing and Mitigating Bias in Medical Artificial Intelligence: The Effects of Race and Ethnicity on a Deep Learning Model for ECG Analysis

As described previously,^8,9 we identified all adult patients (≥18 years of age) with at least 1 digital, standard, 10-second 12-lead ECG and a standard transthoracic echocardiogram obtained within 2 weeks of each other, from January 1994 and February 2017. All ECGs were acquired at a sampling rate of 500 Hz using a GE-Marquette ECG machine (Marquette, WI) and stored using the MUSE data management system for later retrieval. All ECGs were recorded by a trained technician while the patients were supine and still for 10 seconds. We did not perform any preprocessing of the signals. The ECG recording system used 0.15 to 100 Hz filters that resulted in consistent recordings with minimal noise. LV ejection fraction (LVEF) was routinely determined using a hierarchical priority from the most to the least valid method (3-dimensional, volumetrics, biplane, M mode, and visual estimation).¹³ Race and ethnicity were self-reported according the US Census Bureau Office of Management and Budget standards on race and ethnicity categories¹⁴ and recorded in the medical record (non-Hispanic white, Asian, black/African American, Hispanic/Latino, and American Indian/Native Alaskan). Patients who did not report race or ethnicity or who reported multiple categories were not included in the analysis. No other exclusions were applied. The Mayo Clinic Internal Review Board approved waiver of the requirement to obtain informed consent in accordance with 45 CFR 46.116 and waiver of HIPAA (Health Insurance Portability and Accountability Act) authorization in accordance with applicable HIPAA regulations. The data that support the findings of this study are available from the corresponding author on reasonable request.

The primary outcome was the ability of the deep learning network to identify patients with a low LVEF of ≤35% using the ECG signal alone in patients with prior echocardiographic assessments of LV function performed within each racial/ethnic subgroup: non-Hispanic white, Asian, black/African American, Hispanic/Latino, and American Indian/Native Alaskan. Details of the previously described CNN identification of patients with a low LVEF of ≤35% using the ECG signal are included in Appendix I in the Data Supplement. We used the previously published model architecture. The model input was the 12-lead ECG as a matrix of 5000×12 values representing 10 seconds of ECG sampled at 500 Hz. The last layer of the model was a 1×12 convolution filter that combines the data from the different leads. The weights for the lead-specific combining function were learned in the training process. We did not use regularization.

Based on the network output (the probability of the patient having an LVEF ≤35%), we created a receiver operating characteristic curve and AUC as the primary assessment of network strength to diagnose low LVEF in each of the racial subgroups. The threshold probability to define a positive screen for low LVEF was determined from the prior publication on the internal validation set.⁹

We performed 2 secondary analyses. First, we retrained a network using a population restricted to non-Hispanic white patients and tested the algorithm on each of the racial subgroups. Similar to the primary analysis, we created a receiver operating characteristic curve and measured its AUC as primary assessment of network strength to diagnose low LVEF in each of the racial/ethnic subgroups.

Second, a separate neural network was trained on the self-reported race/ethnicity categories to detect ECG features suggestive of an individual’s racial/ethnic subgroup. The network structure of the multiclass network was similar to the left ventricular dysfunction classification network; however, the output layer was replaced by a 5-neuron layer, each reprinting a different racial/ethnic subgroup. During training, the network with the highest overall validation accuracy was selected and testing in the testing data set (the testing data set was not used for any model training and was unseen by the model until model testing phase).

Continuous data are presented as mean±SD and median and interquartile range if highly skewed. For measures of diagnostic performance (AUC, receiver operating characteristic curve, sensitivity, and specificity), 95% CIs were computed under the assumption of independence. Since the current study was aimed at validating the previously derived model, the threshold for determining accuracy, sensitivity, and specificity was selected based on the original work and alerted for any ECG that had an output >25.6% (ie, the ECG indicates at least a 25.6% probability of ejection fraction ≤35%). These routine statistical analyses were conducted using R (version 3.4.2).

The original testing cohort was used for the primary analyses (testing the performance of the CNN across racial/ethnic subgroups), and the original derivation cohort was used in the 2 sensitivity analyses to retrain 2 networks (for race classification and for low LVEF detection using a homogeneous sample), which were then applied to the unseen testing cohort. The testing cohort was a holdout data set and was never used or shown to the model during any stage of the development.

The overall sample included 97 829 patients with paired ECG and echocardiographic data (Figure 1). During the previously published model derivation and testing,⁹ the group was divided into a derivation cohort (n=44 959) and a testing cohort (n=52 870). The breakdown of self-reported race/ethnicity in the testing cohort is shown in Figure 1. Self-reported race was not available in 6587 patients. The majority of patients in the sample were non-Hispanic whites (44 524/46 283, 96.1%).

The baseline characteristics, stratified by self-reported race and ethnicity, are shown in the Table. As shown in Figure 2, the distribution of age and the probability of low LVEF varied by race/ethnicity. The non-Hispanic white patients were slightly older but had a lower proportion of patients with LVEF ≤35%.

The previously derived CNN demonstrated consistent performance to detect low LVEF across a range of racial subgroups in the testing cohort (Figure 3; n=52 870): non-Hispanic white (n=44 524; AUC, 0.931), Asian (n=557; AUC, 0.961), black/African American (n=651; AUC, 0.937), Hispanic/Latino (n=331; AUC, 0.937), and American Indian/Native Alaskan (n=223; AUC, 0.938).

A multiclass neural network (with more balanced representation of the different racial/ethnic groups) was able to discern racial/ethnicity subgroup in the testing cohort with an accuracy of 56.2% (Figure 4). The performance was best for the classification of black/African American (AUC, 0.84), and white, non-Hispanic (AUC, 0.76) race/ethnicity, but less robust for the other subgroups (AUC was between 0.65 and 0.70).

A network trained only on white non-Hispanic individuals from the derivation cohort performed similarly well across a range of racial/ethnic subgroups in the testing cohort with an AUC of at least 0.930 in all racial/ethnic subgroups.

Although the current algorithm was demonstrated to be race invariant in its performance, other algorithms may not be. It will be critical that, as new algorithms are introduced, the performance across a range of demographic subgroups is tested and reported. Only with consistent and rigorous reporting will we be able to determine which algorithms may be less generalizable and require further training before they can be integrated into clinical practice. It is also critical that race and ethnicity data be consistently collected and reported to allow consistency of these algorithms.

We recommend 4 potential strategies to minimize bias in AI. We suggest that (1) investigators remain mindful of the potential for racial/ethnic bias in AI; (2) investigators report performance across a range of demographic subgroups, particularly race/ethnicity in all initial derivation/validation studies; (3) when models are poorly generalizable, investigators perform additional model training on diverse populations (seeking collaborations when needed); and (4) investigators perform external validation in populations that are representative of the population for intended clinical application.

While the number of racial/ethnic minorities in our study was sufficient to calculate an AUC for the model in each group, the sample sizes were still small. Furthermore, the sample was geographically limited to Mayo Clinic, Rochester, MN, which in addition to being a relatively homogenous population of predominantly non-Hispanic whites could have other issues related to geographic, socioeconomic, or healthcare setting generalizability. The algorithm was derived and validated on ECG signals obtained using the GE system, and other healthcare organizations that obtain ECG data through other means may also not be able to apply the algorithm. Lastly, as is true of all deep learning exercises, the specific ECG characteristics used by our unsupervised CNN to classify individuals are not known. We are conducting ongoing studies to create interpretable readouts from the network, but this is not required to apply the current algorithm.

A previously derived CNN had consistent performance for the detection of low LVEF using the ECG across a range of racial/ethnic subgroups. This is not to say that the ECG is race invariant (indeed a separate CNN was able to discern race using ECG features), rather it suggests that the ECG features associated with ventricular dysfunction are race invariant. Other applications of the ECG to risk stratification could be highly sensitive to race. As such, we need to be aware that AI has the potential to exacerbate racial bias and health and healthcare disparities. We recommend vigilance, maintenance of diverse data sets, consistent subgroup reporting, and external validation to ensure responsible use of AI in medicine.