Population Bias in Polygenic Risk Prediction Models for Coronary Artery Disease

There is growing evidence that polygenic risk scores (PRSs) can be applied clinically to improve prediction of individual disease risks.¹ While earlier work on genetic risk scores (GRSs) was based on few variants with genome-wide significant signals for association,^2,3 more recent models included thousands or even millions of genetic variants, which further improved prediction of the risk of coronary artery disease (CAD) and other conditions.^4–6

Common to all GRSs is that the model summarizes the number of risk alleles, weighted by the estimated effects of risk alleles derived from genome-wide association studies. The prediction quality of the score, possibly together with sex, age, and other (clinical) variables, is investigated in a training data set, and the optimum significance threshold, and thus the number of genetic variants used, is selected on the basis of the best performance. Through this, the genomic information of thousands or millions of genetic variants distributed throughout the genome may be compressed into a single value, the (P)GRS. It has been argued that thus summarizing the genetic risk is too simple given the complex biological structure of common diseases. However, we recently found that using PRS is more appropriate than using more complex alternatives with common and widely used machine learning algorithms.⁷ In any case, the generalizability of any GRS then needs to be verified in another independent test data set.⁸

Inouye et al⁴ proposed a meta-analytic-based approach to construct a PRS to predict the risk of CAD using 1.7 million genetic variants. Similarly, Khera et al⁵ introduced a PRS for the risk prediction of 5 common diseases including CAD with 6.6 million genetic variants. Both studies used effect estimates from the CARDIoGRAMplusC4D genome-wide association studies meta-analysis,⁹ and the prediction models were trained on a subset of the UK Biobank (UKB).¹⁰ Khera et al⁵ additionally added age, sex, the first 4 principal components and an indicator variable for the genotyping array in their model, whereas Inouye et al⁴ used the meta PRS by itself. In independent UKB data sets, good predictive performances were replicated (Inouye et al⁴: Harrell C 0.623 [95% CI, 0.615–0.631]; Khera et al⁵: area under the curve [AUC] 0.81 [95% CI, 0.80–0.81]). Risk prediction by PRS was more accurate than that of conventional risk factors, leading the authors to conclude that an individual’s genetic risk of common diseases at birth is predictable and would enable effective prevention or detection strategies.

As a caveat, both author groups pointed out that the proposed PRS were studied largely in individuals of European descent and cannot readily be applied to other ethnic groups without taking into account the target population’s structure.¹¹ Moreover, it is as yet unknown whether the performance of a PRS depends not only on ethnicity but also on smaller genomic differences within a population. In this work, we thus studied the discriminative ability of PRS in data from the UKB and Estonian Biobank (EB)^12–14 as well as data from the German population to test how PRS trained in one European data set perform in other European data sets.

The detailed methods of this work are available as Data Supplement. All included studies were approved by an institutional review committee, and all subjects gave informed consent. Information on the availability of the data that support the findings of this study is available from the corresponding authors of the respective references given in Table I in the Data Supplement. Restrictions apply to the availability of these data, which were used under license for this study. The result data that support the findings of this study are available from the corresponding author upon reasonable request. The code used and the trained PRS models are available at https://github.com/dagola/GO-3269-1-1_code.

The PRS in the UKB and EB data sets and were based on the published imputed genotypes.^10,13 Moreover, we combined six imputed CAD genome-wide association studies from the German population (Table I in the Data Supplement). Randomly chosen subsets of 10 000 individuals each were used for training and the remaining samples as corresponding testing data sets. Since is a case/control data set including smaller numbers of controls as compared to the population-based and , we created an additional data set in which the number of cases is ≈3% as in and . Due to the relative low number of controls in , has a total sample size of 7594 and includes all available controls of . Finally, to test the precision of a PRS trained on a mixed population data set, a combined training data set was used. This included 10 000 individuals with equal numbers of samples from the different population-specific training data sets , and while maintaining the population-specific prevalences. For every PRS, we optimized the hyperparameters (Table 1) in terms of the area under the precision-recall (PR) curve (AUCPR).

Table 2 (upper part) shows that the area under the ROC curve (AUC) in each testing data set is highest when based on the respective model trained in the corresponding population. On the 5% level, these are also significantly better than the respective second-best models, that is, those trained in another European population (DE test data set: DE versus EB: ΔAUC=0.1345 [95% CI, 0.1108–0.1581], P<2.2×10⁻¹⁶; EB test data set: DE versus EB: ΔAUC=0.0409 [95% CI, 0.0238–0.0579], P=2.729×10⁻⁰⁶; UKB test data set: EB versus UKB: ΔAUC=0.009 [95% CI, 0.0047–0.0134], P=4.522×10⁻⁰⁵). The PRS model trained on has slightly but not significantly better performance on the EB and UKB testing data sets compared with the model trained on . The PRS model trained on performs at least as good as the worst population-specific model with a very consistent AUC of about 0.6 in all testing data sets. The performance of the PRS proposed by Khera et al⁵ and Inouye et al⁴ on our testing data sets is added for comparison purposes. Their performances are technically the best on the UKB testing data set (ΔAUC=0.6374 and ΔAUC=0.6377). However, the samples in our testing data set might have an overlap with those used to train the model by Khera et al⁵ and Inouye et al,⁴ and thus are not unbiased estimates. On the PRS model by Khera et al⁵ achieves second-best performance (AUC=0.6699), not significantly worse than that of , and better than the performance on , whereas on its performance is worst (AUC=0.5617). However, the PRS model by Inouye et al⁴ achieves best performance on and worst performance on . Similar results are obtained for the AUCPR. Comparing the distribution of models developed and tested in the UKB and EB data as shown in Figure 1 indicates that there are notable shifts between the different populations. Specifically, many to almost all samples from the UKB data with the highest scores have lower scores than the majority of samples from the EB data. This is also reflected by the estimated CAD prevalence in 100 groups defined by the score percentiles of each population-specific PRS model as shown in Figure 2. Like Khera et al,⁵ we binned individuals into 100 groupings according to the percentile of the GRS, and the unadjusted prevalence of disease within each bin was determined. Here, one would expect higher prevalence of CAD with increasing scores. This is generally true for the PRS evaluated on EB and UKB testing data sets. However, applying the population-specific PRS on other population testing data sets results in inconsistent CAD prevalences, especially at the tails. For example, the models and evaluated on have too high prevalences in the lower percentile groups and too low prevalences in the high percentile groups compared with those of , that is, the extreme scores of nonmatching population-specific PRS do not reflect the subpopulations of very low or high risk. Here, the performance of the PRS models and on (Figure 2, left) are of special note as the estimated prevalences are completely inconsistent.

Given the notable difference in number of SNPs used in each PRS (Table II in the Data Supplement), we additionally compared the performances when fixing the number of SNPs at 2213 genome-wide significant SNPs (P<5×10⁻⁸), the top 3000, 30 000, and 3 000 000 SNPs, respectively. Again, population-specific PRS yielded the highest performances, with the differences even slightly increasing with increasing numbers of SNPs (Figure 3).

We assessed the impact of population-specific data sets of European ancestry on the discriminative performance of PRS and revealed a substantial and clinically relevant drop in performance if training and testing data sets came from different populations. A PRS trained on the combined training data sets performed better than population-specific PRS applied to a different population while being less informative than a population-specific PRS. Importantly, in each of the 3 European populations tested the by far best performance was achieved if the training and testing data set came from the same population. Mimicking the population prevalence in a case/control data set as done for the PRS model trained on did not substantially improve the performance on the testing data sets from different populations. Interestingly, the Khera et al model also performed well on the DE and the Inouye et al⁴ model well on the EB testing data sets, whereas vice versa the predictive values were weaker. While we cannot explain this variability in the data, it is interesting to note that both scores proposed by Khera et al⁵ and Inouye et al⁴ performed best on the UKB testing data, which substantiates our principle findings as it indicates that the population bias effects these scores as well.

Special interest should be payed to models trained on population-based data sets and applied to case/control data sets shown in Figure 2, left. As the estimated prevalences from PRS models and on are inconsistent in contrast to those of applied on and . It appears that training of PRS models on case/control data sets and application on population-based data sets is valid in terms of consistently estimated prevalences. This is likely due to the upscaled fraction of cases in the case/control data set allowing for a better discrimination of cases and controls in any data sets. In contrast, models trained on population-based data sets with a comparatively low fraction of cases do not easily generalize to target data sets with higher fractions of cases. In this case, these models may be too sensitive to detect cases.

It should be noted that subsets of the German and Estonian data sets were part of the CARDIoGRAMplusC4D meta-analysis and thus contributed to the summary statistics used to weight the single SNP contributions in the PRS. Therefore, AUC estimates in the testing data sets from these 2 populations might be inflated.¹⁵

The decreased discrimination performance and shift of scores of population-specific PRS in different populations has direct impact on the clinical utility of risk prediction models by PRS. As scores can be generally lower or higher when applied to samples from other populations than those used for training of the models, estimated risks will also be biased for individuals seeking their personal risk but not matching the population used to derive the PRS and risk prediction models.

Thus, genomic differences between populations must be considered when applying risk estimation models. Importantly, we have shown that this is not only true for individuals from different ethnicities but also for individuals from different populations of the same ethnicity. It is in particular important as the advent of huge biobank data sets tempts to use samples from one biobank only to derive PRS, train risk prediction models and test and validate those models. However, it must be kept in mind that these models may be applicable only to those individuals matching the population structure of the samples in these biobank data sets. Using a mixture of different populations may reduce this bias but will simultaneously also reduce the performance for individuals from the same population. Here, more advanced methods will be needed to maximize the benefit for all. Until then, our conclusion is that each and every single population PRS and population-specific risk estimation model enhanced by PRS will have to be derived on their very own training data set or at least verified for application on the target population, even if PRS or risk models trained in other populations of the same ethnicity are available.

This research has been conducted using the UK Biobank Resource under application number 48012 and the Estonian Biobank Resource, and we thank all participants providing their data for research purposes. Data analyses were performed in part using the High-Performance Computing Center of University of Tartu.