Systematic Evaluation of the Robustness of the Evidence Supporting Current Guidelines on Myocardial Revascularization Using the Fragility Index

Introduction

See Editorial by Seligman et al

RCTs (randomized controlled trials) are considered the gold standard to compare 2 or more treatments. Randomization aims to minimize (and ideally eliminate) the effect of known, unmeasured, or unknown confounders and is traditionally accepted to be able to provide a reliable estimate of whether or not a treatment has an effect (or a larger effect than another).

RCTs are the preferred source of evidence in guidelines and professional societies or expert position papers. They are the foundations of evidence-based medicine and influence the clinical decision-making of the great majority of physicians and affect the outcomes of most of our patients.

The statistical significance of the results of RCTs is generally evaluated using a fixed threshold of metrics, such as the P-value of the CI. However, this approach has been repeatedly criticized by authors and statistical associations in the recent past.^1,2 Indeed, the P-value is fraught with all the limitations inherent to a frequentist statistical framework, which include null hypothesis testing, inability to incorporate prior knowledge, risk of being misinterpreted as proof of evidence or direct probability statement, occasional internal logical inconsistency, accurate estimation only based on large samples, and reliance on approximation for many computational models. In addition, the P-value does not provide any direct information on the existence of a true treatment effect and, according to many, is a simplistic solution to the complexities of probability theory.³P-value-based statistical significance is also heavily affected by methodological limitations and can be lost (or gained) by a shift of few events in one group.

To partially overcome these limitations and to provide an objective estimate of the solidity of RCTs, the fragility index (FI) was introduced in 1990.⁴ The FI is defined as the minimum number of patients whose status would need to switch from nonevent to event to render a significant result nonsignificant. The lower the value of FI, the lower the solidity and robustness of the results.^4,5 While FI is still dependent on the P-value and the problems associated with choosing an arbitrary threshold, it adds additional information as a form of sensitivity analysis which combines the sample size and the precision of the point estimate. Accordingly, FI offers a way to stress-test any study P-values, offering a pragmatic and poignant sensitivity measure and it evidently proves less focus of dichomotous testing in itself but actually weighs it in light of sample size and events accrued. The FI may provide an additional perspective informing practitioners on the robustness of the findings of a RCT analyzed according to a frequentist framework and reporting P-values. Bayesian approaches may provide a better alternative to counter the problems associated with frequentist analyses of randomized trials.

Cardiovascular disease affects >85 million of people in the United States only.⁶ Current guidelines on myocardial revascularization potentially affect the lives of millions of patients worldwide.

In this report, we evaluate the solidity of the evidence supporting the current US and European guidelines on myocardial revascularization using the FI.

Methods

Selection of Randomized Controlled Trials

We identified all RCTs cited in the 2012 American College of Cardiology (ACC)/American Heart Association (AHA) Guideline for the Diagnosis and Management of Patients with Stable Ischemic Heart Disease,⁷ the 2014 Focused Update of the ACC/AHA Guideline for the Diagnosis and Management of Patients with Stable Ischemic Heart Disease,⁸ and the 2018 European Society of Cardiology (ESC)/European Association for Cardio-Thoracic Surgery (EACTS) Guidelines for Myocardial Revascularization.⁹ All trials were independently reviewed by two reviewers (Drs Hameed and Rahouma) and were included for analysis and data extraction if they reported at least one statistically significant dichotomous primary or secondary outcome (P < 0.05 or a 95% CI that excluded the null value). For trials reporting multiple significant primary and secondary outcomes, data were extracted for the primary and secondary outcome with the lowest P-value. The data that support the findings of this study are available from the corresponding author on reasonable request.

Data Extraction

For each RCT, the following data were recorded: citation in European or American guideline, journal of publication and impact factor (according to Thomson Reuters-Clarivate Analytics), year of publication, use or surgical, medical or percutaneous intervention in treatment arm, single- or multicenter study, geographic locations of the participating centers, class of recommendation and level of evidence (LOE) supported. Details of the primary or secondary outcome (definition of outcome, composite or noncomposite end points) sample size, number of events, primary analysis (intention-to-treat or alternatives), statistical power, summary statistics, P-value for the primary outcome, number of patients lost to follow-up (LTF), and number of crossovers were also collected. Two reviewers (Drs Hameed and Rahouma) independently extracted data from the included trials and the first author (Dr Gaudino) resolved any discrepancy.

Calculation of Fragility Index

FI for the statistically significant primary or secondary outcomes were calculated as described by Walsh et al.⁵ The results for each outcome were entered in a 2×2 contingency table following which the P-value for each outcome was calculated using the 2-sided Fisher exact test.

Single participants were iteratively shifted one at a time in the lower-incidence treatment group from nonevent to event and the P-value for the 2×2 table re-calculated. The FI for an outcome equalled the smallest number of participants required to turn the re-calculated P-value nonsignificant (≥0.05).

Calculation of Fragility Index Minus Lost to Follow-Up

FI minus number of patients lost to follow-up (FI-LTF) were calculated following the methods used by Mazzinari et al¹⁰ as the difference between the fragility index and number of patients lost to follow-up.

Statistical Analysis

Continuous variables were reported as medians with their first-third quartile, whereas categorical variables were reported as counts and percentages.

A visual inspection of the data showed that FI and FI-LTF were non-normally distributed; therefore, nonparametric methods were used to compare the groups. The Mann-Whitney U test was used to compare 2 groups, and the Kruskal-Wallis test was used to compare 3 or more groups. To assess the interaction between 2 categorical variables with respect to FI/FI-LTF, the Scheirer-Ray-Hare test was used. Categorical variables were compared using Fisher exact test.

Correlations between FI, FI-LTF, and different variables were calculated using Spearman correlation coefficient for continuous variables and rank-biserial correlation coefficient for dichotomous variables. A local regression curve was used to explore the relationship between the variables.

Multivariable linear regression with a quasi-Poisson distribution was used to explore for independent predictors of FI and FL-LTF, with an exploratory and hypothesis-generating scope. Results are reported as regression coefficient (β) and 95% CI. The 95% CIs were calculated using percentile bootstrapping with 1000 samples. Two-sided significance testing was used and a P-value <0.05 was considered significant. All analyses were performed using SPSS version 22 (IBM, Chicago, IL) and R (version 3.3.3 R Project for Statistical Computing) within RStudio.

Results

Selection of Randomized Controlled Trials and Data Analysis

A total of 414 RCTs were identified: 171 in the ACC/AHA guidelines, and 243 in the ESC/EACTS guidelines. Two hundred eighty-one RCTs reported dichotomous outcomes, of which 167 (64 in the ACC/AHA guidelines and 103 in the ESC/EACTS guidelines) reported at least 1 statistically significant primary or secondary outcome (125 primary, 35 secondary). Seven RCTs were quoted in both ACC/AHA and ESC/EACTS guidelines and were thus entered only once to avoid duplication, leaving 160 RCTs for the analyses (Figure 1; Table I in the Data Supplement). Thirty-three RCTs (20.6%) were published before the year 2000, 69 (43.1%) from 2000 to 2010, and 58 (36.3%) after 2010.

• Download figure
• Download PowerPoint

There were 143 (90.5%) multicenter RCTs, and 78 (48.8%) originated from Europe. The median sample size was 1192 (379–2672). The details of the RCTs and end points are summarized in Table 1.

Details of CORs, LOEs, and numbers of RCTs for the individual guidelines are summarized in Table 2. There were a total of 396 recommendations (161 ACC/AHA, 235 ESC/EACTS) supported by 375 RCTs (151 ACC/AHA, 224 ESC/EACTS): 168 Class I recommendations supported by 168 RCTs; 120 Class IIa recommendations supported by 80 RCTs; 66 Class IIb recommendations supported by 63 RCTs; and 42 Class III recommendations supported by 64 RCTs.

Of these recommendations, 75 were designated LOE A with 178 RCTs; 154 LOE B with 172 RCTs; and 167 LOE C with 25 RCTs.

Fragility Index and Patients Lost to Follow-Up

The distribution of FI, losses to follow-up, and FI-LTF are shown in Figure 2.

• Download figure
• Download PowerPoint

The median FI for all the 160 trials analyzed was 8.0 (95% CI, 5.0–9.0; IQR, 3.0–15.0): 10.0 (95% CI, 8.0–16.0; IQR, 4.0–21.0) for ACC/AHA guidelines, and 5.0 (95% CI, 4.0–8.0; IQR, 2.0–13.5) for ESC/EACTS guidelines (P=0.02). FI was ≤ 3 for 44 RCTs (27.5%): 12 (18.8%) in the ACC/AHA and 34 (33.0%) in the ESC/EACTS guidelines (P=0.07; Table 3).

For primary end points, the overall median FI was 8.5 (IQR, 4.0–16.3): 12.0 (IQR, 5.0–21.0) for ACC/AHA and 7.0 (IQR, 3.0–15.0) for ESC/EACTS guidelines (P=0.04). For secondary end points, the median FI was 4.0 (IQR, 2.0–9.0): 7.0 (IQR, 4.0–22.0) for ACC/AHA and 3.5 (IQR, 1.0–7.0) for ESC/EACTS guidelines (P=0.07; Table 3).

There was no statistically significant difference in FI for era of publication (P=0.66; Table 4), although the median FI was progressively lower for RCTs published in the later study years when considering the eras before 2000, from 2000 to 2010, and after 2010.

There was no difference in the median FI of RCTs used to support different CORs (P=0.25) and LOEs (P=0.16; Table 4).

The FI of RCTs involving surgery in the treatment arm was not significantly different from that of nonsurgical RCTs (P=0.59). RCTs involving percutaneous intervention in the treatment arm had a significantly lower FI than nonpercutaneous intervention RCTs (P<0.01).

RCTs with composite end points had a significantly higher FI than RCTs with single end points (P=0.03; Table 5).

Details of FI-LTF are given in Table 3. The overall median FI-LTF was 1.0 (95% CI, 0.0–3.0; IQR, 0.0–9.0): 3.0 (95% CI, 0.0–7.0; IQR, 0.0–13.0) for ACC/AHA and 1.0 (95% CI, 0.0–3.0;

IQR, 0.0–7.5) for ESC/EACTS guidelines (P=0.45).

FI was lower than the number of patients LTF in 68 (42.5%) RCTs: 28 (43.8%) in ACC/AHA and 45 (43.7%) RCTs in ESC/EACTs guidelines (P=1.00; Table 3; Figure 2).

Correlation Between Fragility Index, Fragility Index Minus Number Lost to Follow-Up, and Trial Characteristics

The FI was significantly correlated with the sample size (Spearman correlation [R]=0.35; P<0.001), number of events (R=0.60; P<0.001), statistical power (R=0.21; P=0.02), impact factor of journal of publication (R=0.34; P<0.001), use of intention to treat analysis (R=0.20; P=0.01), and of composite end points (R=0.17; P=0.03). FI was negatively correlated with use of percutaneous intervention in the treatment arm (R=−0.25; P=0.001), the P-value level (R=−0.63; P<0.001), and was lower for primary end points (R=−0.18; P=0.02; Figure 3; Table II in the Data Supplement).

• Download figure
• Download PowerPoint

The FI-LTF was negatively correlated with the P-value level (R=−0.42; P<0.001) and the number of patients lost to follow-up (R=−0.65; P<0.001; Table II in the Data Supplement).

Multiple regression revealed that sample size, number of events, and P-value level were independent predictors of FI (P<0.01), while sample size, number of events, number lost to follow-up, and P-value level were independent predictors of FI-LTF (P<0.05; Table III in the Data Supplement).

Discussion

We have found that the RCTs supporting the current guidelines on myocardial revascularization are generally fragile. The median FI was 8, meaning that a shift of 8 patients from nonevent to event would have significantly changed the results of the original analysis. Of note, 27.5% of the RCTs had a FI≤3, and even the median FI of RCTs used to support class I LOE A recommendations (considered the most solid in clinical medicine) was <10. Even more concerning, the median FI-LTF was 1 and 42.5% of RCTs had a negative FI-LTF value, indicating that their FI was smaller than their number of patients who were lost to follow-up.

Despite the common critiques of surgical RCTs, trials evaluating surgical interventions had FI similar to nonsurgical trials. In contrast, trials aimed at evaluating percutaneous treatment had significantly lower FI.

Confirming previous reports,¹⁰ the sample size and the impact factor of the journal of publication were significantly correlated with the FI. The number of events, statistical power of the trial, P-value level, use of composite end points and intention to treat analysis were also correlated with the FI. Similar to previous findings in other fields,^11,12 the treatment effect size was not correlated with the FI.

Interestingly, the FI decreased (although not significantly) over time, with most recent RCTs having lower FI. This may be related to the increasing difficulty in finding support for large RCTs and may elicit concerns regarding enacted policy changes that may have been based on fragile evidence.

Traditionally, the statistical significance of an RCT is judged based on a fixed threshold of P-value (usually 0.05). The P-value level is influenced by methodological factors and may vary substantially with shifting of only few events from one group to the other. In fact, the use of the P-value approach has been heavily criticized in recent years. The American Statistical Association, in a statement of statistical significance and P-values, has summarized the many issues related to the use of boundary P-values.² Reliance on a fixed P-value level has also been identified as one of the possible cause of the low level of replication rate in current scientific research.

The FI was introduced in 1990 with the aim of complementing the P-value and providing an intuitive measure of the solidity of the results of an RCT.⁴ The FI is the number of participants that need to switch from nonevent to event in the lower incidence treatment group for a trial to lose statistical significance.⁵ In a rather misnomer fashion, a lower FI value indicates lower solidity of the results. An additional important measure is the FI-LTF, as one can make the case that the outcome of patients LTF could have changed the statistical results of the trial. This is particularly relevant when the likelihood of occurrence of the outcome may be the reason for the LTF.¹³ So far, no defined FI boundaries exist for the definition of frail RCTs. Intuitively, the FI must be correlated with the sample size, treatment effect, power and existing evidence, and the FI threshold may vary from case to case.

Walsh et al⁵ found that among 399 RCTs published in high-impact medical journals between 2004 and 2010, the median FI was 8 and that 25% of them had a FI ≤3. Notably, in 53% of the trials, the FI was lower than the number of patients LTF. Docherty et al,¹¹ in a review of the RCTs supporting the guidelines for the management of patients with chronic heart failure, had more reassuring results with a median FI of 26, 35% of the trials with a FI≤10 and 20% with FI lower than the number of LTF.

In general, the strength of the published RCTs in different fields has been reported to be low. In a review of the RCTs quoted in the 2016 Chest Guidelines and Expert Recommendations for Venous Thrombo-embolism, the median FI was 5 (median sample size, 400), while a similar review of the 2017 guidelines for the treatment of diabetes mellitus reported a median FI of 16 (median sample size, 2548). Low or very low FI have been reported in trauma (median FI, 3; median sample size, 207), critical care (median FI, 2; median sample size, 126.5), nephrology (median FI, 3; median sample size, 134), spine and sport surgery (median FI, 2 for both, median sample size, 132 and 64, respectively), and anesthesiology (median FI, 4; median sample size, 150).^12,14–20

It is important to note that there are ethical reasons to design RCT sample sizes to produce the required level of evidence using the minimum number of participants. In fact, while enrolling a larger number of participants may produce stronger evidence against the null, this implies randomizing patients when some level of evidence could be already generated from the available data, and may be seen as violating the equipoise principle. On the other hand, fragile results that are contradicted by subsequent studies or require confirmation in other trials may also be potentially harmful to patients and elicit equally important ethical questions. The delicate balance between the number of patients whose treatment is based on randomization and the solidity of the achieved results is the basis of RCT sample size calculation.

Our findings and similar ones from other medical and surgical specialties highlight the need, whenever using an RCT to inform practice guidelines, to carefully consider, on top of clinical and statistical significance, the actual robustness of the results, and the consequent role of play of chance in guiding its conclusions. We suggest that future practice guidelines provide detailed reports of RCT fragility, and that fragility should be considered when planning pivotal and pragmatic RCTs, and its measurements consistently provided in the main publications to inform patients, clinicians, and stakeholders.

Several limitations of this analysis need to be considered. The FI, like the P-value, should not be interpreted as a measure of the effect, but only of the fragility of the results of a trial. In addition, the FI does not per se overcome the limitation of a frequentist framework, but simply provides an additional perspective on a study weaknesses. There are no established boundaries to define an RCT outcome as robust or fragile and the same FI can have different meanings in different clinical contexts. It is also plausible that the FI of primary outcomes consisting of hard end points, such as mortality, may warrant a different interpretation than those involving functional measures. As trials are usually powered for the primary outcome, calculation of the FI for secondary outcomes, which should only be considered hypothesis generating, must be viewed with skepticism. FI, P-values, events, and sample are mathematically related and thus their multivariable analysis is limited by collinearity and clustering features, with eventual results mainly exploratory and hypothesis generating. Also, the FI can be applied only to trials with a positive result and a dichotomous outcomes and its calculation convert time-to-event into dichotomous outcomes. Finally, only 160 RCTs were eligible for analysis and the generalizability of our results may be limited.

In conclusion, we have found that the solidity of the RCTs used to support the current guidelines on myocardial revascularization is low and seems to be decreasing in the most recent years. Our data support the need for large RCTs addressing important clinical questions with adequate trial design, power, and sample size. Our findings also suggest that metrics related to FI of the supporting evidence should be incorporated in the recommendations proposed by professional guidelines.

Acknowledgments

Drs Gaudino, Hameed, Biondi-Zoccai, Tam, Rahouma, Khan, Angiolillo, Benedetto, Taggart, Girardi, Crea, MR, and S. Gerry are experts in clinical research on myocardial revascularization. Dr Biondi-Zoccai and S. Gerry are experts in statistical analysis. Drs Gaudino, Hameed, Biondi-Zoccai, Tam, S. Gerry, MR and Drs Khan, Angiolillo, Taggart, Girardi, Crea, MR, and S. Gerry have expertise in clinical research methodology.

Footnotes