Fragility Index in Cardiovascular Randomized Controlled Trials

Introduction

See Editorial by Seligman et al

Randomized controlled trials (RCTs) are considered the most reliable source of evidence by which to guide clinical care.¹ Therefore, it is critical for a trial to report results in a manner that is easy to interpret. Summary estimates are traditionally reported with 95% confidence intervals, whereby a P≤0.05 is considered to be of sufficient statistical significance to reject the null hypothesis. However, in some cases, a different P threshold might also be used. Nevertheless, such statistics are subject to misinterpretation, especially when event rates and sample sizes are small and follow-up times are limited.^2,3 Multiple factors such as power of the trial to detect outcome differences, overall event rates, patient loss to follow-up, and the subject’s exposure time to the intervention for a relatively rare outcome to emerge can greatly affect the P value.^4,5 The fragility index (FI) has emerged as an innovative tool to ease interpretation of statistical findings, which may provide additional value over the commonly reported P value, risk reductions, and confidence interval.^6,7 However, nondichotomous outcomes cannot be assessed using this index. The FI is defined as the minimum number of patients who must be moved from the nonevent group to the event group to turn a significant result to a nonsignificant one.¹ While the FI of trials in multiple specialties has been assessed, individual trials in all fields including cardiovascular medicine have been slow to adopt the use of FI.^6–13 To fill this knowledge gap, we conducted this study to assess the FI of cardiovascular RCTs published in high-impact journals.

Methods

The authors declare that all supporting data are available within the article and its Data Supplement.

Data Sources

In this retrospective analysis, we conducted a Medline search using the search specification: (“Lancet (London, England)”[Journal] OR “The New England journal of medicine”[Journal]) OR “Journal of the American College of Cardiology”[Journal]) OR “Journal of the American Medical Association”[Journal] OR “Circulation”[Journal] OR “European heart journal”[Journal] AND (Randomized Controlled Trial[ptyp] AND (“2007/01/01”[PDAT]: “2017/12/31”[PDAT])). No search restrictions were applied. Since publicly available data were utilized, institutional review board approval was not applicable.

Journal Selection

Journals were selected for the present study based on a combination of the following features: impact factor, range of circulation of studies, specialization in publication of cardiovascular RCTs, and global recognition for consistent publication of influential RCTs over the last several decades. The New England Journal of Medicine, The Lancet and Journal of the American Medical Association were selected for having the highest impact factors in General Medicine, while Journal of the American College of Cardiology, European Heart Journal, and Circulation were selected for having the highest impact factors in the field of cardiovascular medicine and being acclaimed for focusing primarily on the publication of cardiovascular studies.

Study Selection

All RCTs were assessed for inclusion from the 3 cardiovascular journals, namely Circulation, Journal of the American College of Cardiology, and European Heart Journal. RCTs from The Lancet, New England Journal of Medicine, and Journal of the American Medical Association, the 3 non-cardiovascular journals, were first screened at abstract and title level for determination of possible cardiovascular nature. RCTs which mentioned an intervention, outcome, or recruited sample population as being cardiovascular (related to any of the following cardiovascular clinical subspecialties: heart failure, interventional cardiology, preventive cardiology, electrophysiology, cardiac imaging, and other) or related to stroke, anywhere in the title or abstract, were included.

The following eligibility criteria were applied: (1) phase 3 or 4 RCTs studying a cardiovascular intervention or outcome; (2) sample size ≥500 patients (an arbitrary cutoff to delineate larger trials that are more likely to impact practice; (3) parallel arm study design; (4) at least 1 statistically significant binary outcome or time-to-event outcome. At abstract and title level screening, letters, editorials, systematic reviews/meta-analyses, opinions, observational studies, economic/cost effective analyses of RCTs, cohort nonrandomized studies, quasi-randomized trials, and post hoc/secondary analyses of previously reported RCTs were excluded.

Data Extraction

Two reviewers (Drs Ochani and Shaikh) independently screened all the RCT abstracts based on a priori eligibility criteria. Data were extracted on a prespecified data collection form, and discrepancies were resolved by referring to a third investigator (Dr Khan). The data collection form focused on details of statistically significant primary outcomes, including outcome type, primary study outcome, event rates and sample sizes of comparative groups, number of participants lost to follow-up, and follow-up duration of each trial. Following the statistical hierarchy, prespecified primary outcome was given preference if a trial had reported multiple outcomes as statistically significant.

Additional data were also extracted for the location of the trial (Asia, Europe, North America, multiple countries, and other), blinding (double-blind, unblinded, single-blinded), centers (single center, multicenter), type of intervention (pharmaceutical, surgical and other), control (placebo-controlled, active comparator), type of funding (government or private), and whether the trial was conducted on intention to treat principle.

Statistical Analysis

The Statistical Package for the Social Sciences (v.23, International Business Machines Corporation, New York, NY) was used to conduct the analysis. Data from each trial were presented on a 2-by-2 contingency table, and the FI was calculated in the manner described by Walsh et al.⁶ For time-to-event outcomes, total number of events in each group over the entire follow-up time was included.⁶ Events were added to the smaller event group and nonevents were simultaneously subtracted, while maintaining a constant patient population. The Fisher exact test was then used to recalculate the 2-sided P value, while iteratively adding of events until the P value reached or exceeded 0.05. The number of additional events required to reach a P of ≥0.05 was defined as the FI. Loss to follow-up was compared with FI for each trial, as it affects both the number of study participants at risk and the number of recorded events.¹ We reported the overall FI, FIs by subgroups (clinical subspecialty, year of publication and country of origin, overall sample sizes, and event rates per study outcome) as medians with their interquartile ranges. We used the Pearson correlation to assess for a relationship between FI and impact factor. The Kruskal-Wallis test was applied to find possible statistically significant relationships between FI and nominal variables when there were >2 groups as opposed to the Mann-Whitney U test when there were 2 groups. In addition, we also used the Pearson correlation to assess for a relationship between sample size of the RCT and FI. Last, we conducted a descriptive analysis of additional data that were collected and compared frequencies for each item by journal and median FIs of those journals. The FI is an absolute measure, and it does not account for the sample size; thereby making it difficult to compare fragility of different RCTs or set a standard value that can be defined as fragile results. To overcome this, we calculated the fragility quotient (FQ), as previously reported in the literature,^14–17 which is the FI divided by the sample size. This would enable us to see what proportion of events must be moved to nonevents to make the results nonsignificant.

Results

Figure 1 shows the detailed literature search process. Out of 4994 search results, 123 RCTs met the eligibility criteria (Data Supplement). The Table displays the number of RCTs in each subgroup, along with median FIs. The range of sample size in the analyzed trials was 500 to 50 156. Of the 123 RCTs, 59 (48%) were published in New England Journal of Medicine, 14 (11.4%) in Journal of the American College of Cardiology, 24 (19.5%) in The Lancet, 9 (7.3%) in Journal of the American Medical Association, 12 (9.8%) in Circulation, and 5 (4.1%) in European Heart Journal. All included trials used a P of ≤0.05 as the threshold for significance.

• Download figure
• Download PowerPoint

The median FI was 13 (5–26). The median total sample size was 2466 (1005–7513), of which 1035 (485–3264) were controls and 1043 (499–3241) were interventions. The median total event rate was 241 (92–605), 128 (51–355) for controls, and 102 (40–262) for interventions. The median follow-up time was 12 months (3.5–36), while the median loss to follow-up was 17 (3–32.5).

Figure 2 shows the number of RCTs within each range of FI. We observed that 13% of the RCTs (n=16) had an FI between 1 and 2, 9.8% (n=12) between 2 and 4, 17.1% (n=21) between 4 and 8, 17.1% (n=21) between 8 and 16, 22.8% (n=28) between 16 and 32, 8.9% (n=11) between 32 and 64, 9.8% (n=12) between 64 and 128, while 1.6% (n=2) had FI between 128 and 256. In 37 RCTs (30.1%), patients lost to follow-up were greater than the FI. In the studies with loss to follow-up greater than the FI, the median FI was 8 (3.5–15.5) and the median loss to follow-up was 29 (17–115.5).

• Download figure
• Download PowerPoint

FI and impact factor (r=−0.064; P=0.485) were not statistically related; however, FI and sample size demonstrated a weak statistically significant positive correlation (r=0.324; P<0.001; Figure 3). The median FQ was 0.0042 (0.0020–0.0110). The median FQ of studies with a FI lesser than loss to follow-up was 0.0031 (0.0010–0.0072), while the median of FQ of the remaining studies was 0.0050 (0.0023–0.0127).

• Download figure
• Download PowerPoint

We found no differences in the FI among journals: FI was 21 (10–27.5) for European Heart Journal, 17 (6–34) for New England Journal of Medicine, 12.5 (2.25–23.5) for Circulation, 11.5 (5–21.5) for The Lancet, 7.5 (4.75–23) for Journal of the American College of Cardiology, and 8 (2.5–19) for Journal of the American Medical Association. These differences did not demonstrate statistical significance (P=0.440). Multiregional RCTs were the most robust, FI=22 (12–53.25). Trials conducted exclusively in Asia and other countries had findings that were as robust (FI=11 [4.25–25.25] and FI=11 [4.5–21.5], respectively) as trials conducted in United States (FI=11 [4.25–35]; P=0.023).

Results of RCTs based on study design, double-blind, single-blind and unblinded, did not differ in degree of robustness (FI=19 [5.5–56.5], FI=8 [5.75–21.25] and FI=9 [4.75–18.5], respectively; P=0.097). Locations where RCTs were conducted, multicenter or single center, did not have an impact on the degree of their robustness (FI=14.5 [5–28.25], FI=8 [3–19], respectively; P=0.170). The type of the trial, placebo controlled or active-comparator, also did not predict a difference in robustness (FI=13.5 [5–31.5] and FI=12 [5–25], respectively; P=0.747).

Intervention wise, trials reporting surgical interventions (FI=8 [4.5–13]), and other interventions, such as offering financial incentives, etc (FI=7 [3–20]) had less robust results, whereas RCTs with pharmaceutical interventions were considerably more robust, FI=19 (7–52; P=0.002).

Clinical subspecialty analysis yielded substantial differences in FIs. Studies in preventive cardiology had the most robust results (FI=19 [8.5–43]), followed by electrophysiology (FI=12 [7–22]), heart failure (FI=11 [3.25–51.5]), interventional cardiology (FI=8 [5–19]), and other (FI=3 [1.75–13.25]; P=0.020).

Discussion

Main Findings

In a cross-sectional view of positive studies published in top-tier medicine and cardiovascular journals, we observed considerable variation in the robustness of clinical findings. Our 2 main findings were that nearly 1 in 4 studies had a low FI (between 1 and 4), and in 37 trials (30.1%) the number of patients lost to follow-up was higher than the FI.

Our observed median FI among cardiovascular trials was higher than those previously reported in other subspecialties, including critical care medicine,⁸ spinal surgery,⁹ sports medicine,¹⁰ and anesthesiology,¹² all of which had FIs <5. The number of cardiovascular trials in which patients lost to follow-up exceeds the FI also compares favorably to studies from other fields, which had trials with loss to follow-up >40% to 70%.^6,9,11,13

Although it is encouraging that cardiovascular trials compare well to trials in other fields,¹⁸ there is room for improvement in the design of cardiovascular trials, considering the validity of outcomes can be disturbed by a mere change of a median ≈13 events across these trials.¹³ Furthermore, the FQ was relatively low (0.0042); meaning that the study’s outcome would be nonsignificant if just 0.4 patients per 100 experienced a different event. Statistical significance, in the form of simple P value, can be misleading and may represent a disconnect with clinical impact of the intervention.¹⁹ While experienced clinicians and researchers may have a more in-depth interpretation of the results, some readers likely rely on P value, as the P value remains the primary value around which investigators base their conclusions.²⁰ Furthermore, some readers may not be familiar with the interpretation of other reported statistics such as power of the trial to detect estimate differences, event rates, patient loss to follow-up, and follow-up duration (allowing a sufficient time window for outcome differences to emerge), which can greatly affect the statistical significance of the results.^4,5

Examples of Fragile Results in Cardiovascular Trials

McIntyre et al²¹ provide a good example of statistically fragile results when considering the most recent trials of patent foramen ovale closure versus medical therapy for the secondary prevention of cryptogenic stroke. Two of the 3 positive trials (REDUCE [Patent Foramen Ovale Closure or Antiplatelet Therapy for Cryptogenic Stroke] and RESPECT [Randomized Evaluation of Recurrent Stroke Comparing PFO Closure to Established Current Standard of Care Treatment]) had fragility indices less than the number of patients lost to follow-up. In the long-term publication of RESPECT, the FI for the primary end point was 1, but there were 56 (11%) patients lost to follow-up in the active arm and 67 (14%) of patients lost to follow-up in the control arm. Similarly, in REDUCE, the 4% absolute risk reduction in the primary end point in the active arm came with a FI of 5, which is considerably lower than the 22 total patients lost to follow-up in both arms.^22,23

In the CSPPT (China Stroke Primary Prevention Trial), which enrolled 20 702 participants with a history of hypertension and no previous cardiac events, Huo et al found that the addition of 0.8 mg folic acid to a regimen of 10 mg enalapril significantly reduced the risk of the primary outcome (first stroke) by 0.7% (2.7% versus 3.4%). The FI of this trial was calculated as 23. However, the numbers lost to follow-up (n=68), were almost 3× greater than the FI.²⁴

The ARISTOTLE trial (Apixaban for Reduction in Stroke and Other Thromboembolic Events in Atrial Fibrillation; 18 201 participants) discovered that a twice-daily dose of 5 mg apixaban was superior to warfarin (target international normalized ratio, 2.0–3.0) in significantly reducing the primary outcome of hemorrhagic stroke, ischemic stroke, or systemic embolism in atrial fibrillation by 0.33% per year (1.27% versus 1.60%). The FI of this trial calculates as 12; however, the number of subjects lost to follow-up (69) exceed the number of events needed to change the results from significant to nonsignificant by almost 6 times.²⁵

We found no statistically significant associations between the FI and journal of publication, its impact factor, or the year of publication. This suggests that cardiovascular RCTs published in top-tier journals over the last decade have similar degrees of robustness. We did observe higher fragility indices from trials performed on many continents and trials within the subspecialty of preventive cardiology.

Our findings compare well to a previous trial from Docherty et al²⁶ who assessed the FI of clinical trials underpinning heart failure guidelines. Their analysis, which included trials from the 1980s, found a median FI of 26 (range 0–118), an FI of <10 in 35% of trials, and FI less than the number lost to follow-up in 20% of trials.

Practical Implications

The FI is an easy to calculate index that any clinician can use. It provides readers with an alternative way to evaluate robustness of trial results in addition to the P value and effect size estimates. In 2016, the American Statistical Association released a statement on the context, process, and purpose of P values.²⁷ Their concluding sentence was that “no single index should substitute for scientific reasoning.”

Thus, the FI and its comparison to the numbers of patients lost to follow-up offers users of the medical evidence (patients and clinicians alike) another way to assess the efficacy and safety of a therapeutic intervention. Knowing that the number of patients lost to follow-up exceeds the number of events needed to turn a pivotal trial to statistically insignificant results can play an important factor in decision making. Shared decision-making tools could incorporate the FI as a way to assist in conveying the level of certainty, when trading benefits and harms.²⁸ It must be remembered that a trial with a high FI does not necessarily indicate a robust result—if the loss to follow-up exceeds FI, the results still need to be interpreted with caution. Similarly, it is crucial to realize that a low FI does not mean that the results of an RCT be considered as trivial. RCTs are performed with finite resources and are thus constructed to maintain an equilibrium between sample size and anticipated efficacy. This will, by design, lower the FI making the results depend on fewer events. Multiple factors including trial design and elimination of biases should also be taken into account, in addition to the FI, when interpreting results.²⁹

Limitations

Our study has limitations. First, we only included trials that were of 2-by-2 factorial design. We were not able to evaluate RCTs with primary end points that were ordinal or continuous in nature. Thus, our analysis could miss some important cardiovascular RCTs. Therefore, if the FI is used on a wider scale, many studies will not be able to utilize it. Second, time-to-event outcomes were included in the analysis. While multiple studies have included such outcomes in the calculation of the FI,^6,30 the FI does not account for the contribution of time to the difference in treatment effects.³¹Third, our analysis did not take allocation concealment into account, and hence, any impact it had on FIs could not be evaluated. Last, we did not analyze median FIs of similar and differing intervention strategies and thus could not adequately establish whether actually comparing like-with-like end points yields more accurate differences in FIs.

Conclusions

Although cardiovascular RCTs may be statistically significant based on the P value (P≤0.05), the results are often fragile, and the statistical significance of their findings may hinge on a small number of events. Therefore, when evaluating findings of trials, especially ones with large numbers of patients lost to follow-up and those in which patients lost to follow-up exceed the FI of that trial, clinicians should be cautious when translating results to bedside practice.

Footnotes