Bias and Loss to Follow‐Up in Cardiovascular Randomized Trials: A Systematic Review

Background Loss to follow‐up (LTFU) is common in randomized controlled trials. However, its potential impact on primary outcomes from cardiovascular randomized controlled trials is not known. Methods and Results We conducted a prospective systematic review (PROSPERO: CRD42019121959) for randomized controlled trials published in 8 leading journals over 5 years from January 2014 to December 2018. Extent, reporting, and handling of LTFU data were recorded, and the proportion of a trial's primary outcome results that lose statistical significance was calculated after making plausible assumptions for the intervention and control arms. These assumptions could drive differential treatment effects between the groups considering relative event incidence between LTFU participants and those included in the primary outcome. We identified 117 randomized controlled trials of which 91 (78%) trials reported LTFU, 23 (20%) reported no LTFU, and 3 (3%) trials did not report on whether LTFU occurred. The median percentage of study participants lost to follow‐up was 2% (interquartile range, 0.33%–5.3%). Only 10 trials (9%) had a low cluster of risk factors for impairment in trial quality. The percentage of trials losing statistical significance varied from 2% when the relative event incidence for LTFU between the randomized groups was 1 for the intervention arm and 1.5 for the control arm to 16% when the relative event incidence was 3 for the intervention arm and 1 for the control arm. Conclusions Almost 1 in 6 (16%) cardiovascular randomized trials published in leading journals may have a change in the primary outcome if plausible assumptions are made about differential event rates of participants lost to follow up. There is scope for improvement arising from LTFU in randomized trials in cardiovascular medicine. Registration URL: https://www.crd.york.ac.uk/prospero; Unique identifier: CRD42019121959.

participants associate with clinical events, then bias can arise. This is particularly relevant in open-label studies in which assessors know the group allocations of the participants. Loss to follow-up (LTFU) in this scenario could favor the intervention arm and neutralize the benefit of randomization. 10 It is plausible that attrition bias associated with LTFU drives either overestimation or underestimation of treatment effects. 11,12 Classification of LTFU and recommendations for dealing with LTFU have been made. 13 Crucially, however, the contemporary prevalence and effects of LTFU within cardiovascular trials is not known. This prospective systematic review and meta-analysis was designed to analyze the prevalence and potential impact of LTFU in cardiovascular RCTs. The primary aim was to assess the proportion of trials in which the primary efficacy end point would change if plausible assumptions were made about participants who were unaccounted for in the original analysis. In addition, we assessed estimates of treatment effect according to the extent, reporting, and handling of LTFU and trial characteristics associated with LTFU.

METHODS Eligibility
All supporting data are available within the article and its online supplementary files. Ethics approval was not required. We predefined reports as being eligible for inclusion in this analysis if an RCT in cardiovascular disease was described and published in one of the 5 leading general medical journals and 3 cardiology journals with the highest impact factors (Annals of Internal Medicine, BMJ, JAMA, Lancet, New England Journal of Medicine, Circulation, European Heart Journal, and Journal of the American College of Cardiology). A 5year publication period was set from 2014 to 2018. An additional inclusion criterion was if a patient-important binary primary outcome was statistically significant at a 2-sided α of 0.05. The rationale behind focusing on statistically significant trials in major journals only is that the results of these trials are most likely to influence clinical guidelines. Therefore, a change in significance of a risk ratio due to bias might affect patient care to an important extent. Cluster trials, crossover trials, Nof-1 trials, and trials reported in research letters were excluded. Equivalence and noninferiority studies were excluded unless the authors prespecified testing for superiority. Reports describing secondary analyses of randomized trials were excluded.
A patient-important outcome was defined as an outcome that would be undesirable for a patient to experience and the patient would thus try to prevent it by undergoing an effective treatment. Mortality and morbidity are examples of outcomes that were included. Surrogate outcomes were considered as nonpatient important (Data S1). The protocol was registered on PROSPERO (CRD42019121959).

Literature Search
Reports of RCTs were identified from Medline and Embase using OVID (Data S2). The search was restricted to clinical RCTs in cardiovascular disease published in the selected journals between 2014 and 2018. Trials were considered statistically significant if the 2-sided 95% CI of an estimate of the relative risk did not include 1.0 or if the 2-sided P value for superiority was <0.05 when no CI was reported. A calibration exercise was performed before the search. One reviewer identified and reviewed the potentially eligible reports based on an agreed screening form (Data S3). The list of included and excluded reports was provided to the 2-person reviewer team after screening. Disagreements were resolved by consensus, with the assistance of a third reviewer as required.

CLINICAL PERSPECTIVE
What Is New?
• More than three quarters of cardiovascular randomized controlled trials have participants who are lost to follow-up. Statistical handling of these data vary widely. • Up to 1 in 6 trials may have a change in the primary outcome if plausible assumptions are made about differential event rates of participants lost to follow up.
What Are the Clinical Implications?
• In dealing with loss to follow-up (LTFU), prevention should be prioritized; otherwise, estimation can be made by using the worst assumption. • When reporting LTFU, authors should provide baseline characteristics of LTFU participants, extent of follow-up before exclusion, and time of dropout and should address implication of LTFU when interpreting results. • Inadequate allocation concealment is an independent factor associated with LTFU and may drive differential treatment effects.

Statistical Analysis
The analysis is explained in more detail in the online supplement (Data S5). Methodological and reporting quality of the included trials was assessed, as suggested by Bikdeli et al 14 and the Cochrane risk-of-bias assessment tool. 16 The extent of LTFU was calculated as the percentage of LTFU in each trial from each arm (intervention and control). The ratio of LTFU rate to primary event rate was also reported. A univariable random-effects metaregression analysis was conducted using the log odds of participants lost to follow-up as the dependent variable and general trial characteristics and methodological characteristics as independent variables. The potential impact of LTFU on the primary outcome analysis was evaluated by making assumptions about the outcomes in LTFU participants (Data S6). An estimation algorithm proposed by Akl et al 17 was adopted with relative incidence of outcomes in LTFU patients compared with patients who were followed-up (RI LTFU/FU ), ranging from 1 to 3. In addition, the following common assumptions were used for calculations: none of the participants lost to follow-up had the event; all participants lost to follow-up had the event; none of those lost to follow-up in the treatment group had the event, and all those lost to follow-up in the control group did (best case scenario); all participants lost to follow-up in the treatment group had the event and none of those in the control group did (worst-case scenario).
*General cardiology trials in this review referred to pharmacological trials and lifestyle-changing trials. † Number shown refers to trials that did the following. ‡ Complete case analysis is defined as an analysis that only include patients with complete outcome data. LTFU patients are excluded from the whole analysis.  (54) No blinding of patients † 76 (65) Early stop 9 (8) Not using intention-to-treat analysis ‡ 29 (25) Absence of protocol § 31 (26) Without explicit statement about status of LTFU 43 (36) LTFU indicates loss to follow-up. *Allocation concealment defined as to the person enrolling participants does not know in advance which treatment the next person will get which usually involves the use of computer algorithms. It seeks to prevent selection bias by protecting the assignment sequence until allocation, and can always be successfully implemented. 136 It is considered to be adequate according to the definition reported by Jüni et al. 3 † Blinding defined as to the withholding information about the assigned interventions from people involved in the trial who may potentially be influenced by this knowledge; blinding is performed to prevent performance and ascertainment bias by protecting the sequence after allocation and cannot always be implemented. 136,137 It is considered to be adequate only if clearly indicated. ‡ Intention to treat analysis defined as an analysis that included all randomized participants in the analysis who are all retained in the group to which they were allocated. 3,136 § Consider as absence if the protocol is not published before or is included as appendix beside the main report. Figure 2. Distribution of trials according to methodological and reporting quality assessments that might impair the outcomes of the trial. Distribution of trials according to the number of methodological and reporting quality characteristics (methodological confounders) they possess after the assessment: 9% of the trials had none of the methodological confounders (n=10), 21% of the trials possessed 1 methodological confounder (n=24), 32% of the trials possessed 2 major methodological confounders (n=38), and 26% of the trials possessed 3 major methodological confounders (n=30). In addition, 12% of the trials had >3 methodological confounders. This list of methodological confounders analyzed included the following: (1) inadequate allocation sequence concealment, (2) no blinding of patients, (3) early stop of trial, (4) not using intention-to-treat analysis, (5) absence of protocol, and (6) no explicit statement about status of loss to follow-up.

RESULTS
After excluding duplicates and screening for eligibility, 117 studies were included from a total of 3668 from the initial search ( Figure 1). The list of the 117 studies included in this analysis is provided in Table S1.  The mean age of 407 229 study participants was 64.2 years (30% female). The trial subspecialties were electrophysiology (19%) heart failure (3%), interventional cardiology (28%), cardiac surgery (3%), general cardiology (44%), and cardiovascular imaging (3%). Baseline study characteristics of the included trials are reported in Table 1.

Assessment of the Methodological Quality of the Trials
The analytical methods that were used for handling LTFU in the primary analysis of the included trials are presented in Table 1. The most commonly used method was censoring at time of LTFU in a time-to-event analysis (N=75; 64%). Two trials (2%) assumed that no LTFU participants experienced events, whereas 10 (8%) used complete case analysis and 2 (2%) used a worst-case scenario in which only the control arm had events. Two trials (2%) used multiple imputation, whereas none reported using inverse probability weighting.
Regarding the reporting of LTFU, 85 (73%) used a Consolidated Standards of Reporting Trials (CONSORT) diagram. Seventy (77%) trials reported that LTFU occurred in the intervention and control arms separately. However, none of the trials compared baseline characteristics of LTFU participants with followed-up participants. The implications of LTFU are discussed in 6 trials (7%). Table 2 and Figure 2 demonstrated the number of trials meeting the characteristics (methodological confounders) for impairment in the quality of trial design. Allocation concealment was adequate in 54 trials (46%). Patients were blinded adequately in 41 trials (35%). In 9 trials (8%), enrollment was discontinued prematurely. Twenty-nine trials used an intention-to-treat analysis (25%). Thirty-one trials (26%) provided a protocol. Forty-three trials (36%) did not state the status of LTFU explicitly in the report. Only 10 trials (9%) were free from any methodological confounders that might impair the methodological quality.
Random-effects metaregression analysis (  Among the 91 trials, percentages of results that would lose significance under less plausible assumptions: (1) none of the LTFU participants had the event, 4%; (2) all the LTFU participants had the event, 11%; (3) none of those lost to follow-up in the treatment group had the event, and all those lost to follow-up in the control group did (best case scenario), 3%; (4) all participants lost to follow-up in the treatment group had the event, and none of those in the control group did (worst case scenario), 33%. FU indicates follow-up; LTFU, loss to follow-up.
*RI LTFU/FU is the relative event incidence among those with LTFU compared with those followed up. Assumptions being made toward the outcome of LTFU in each trial from the search and the subsequent calculation made. In total, 117 trials from 8 journals covering 407 229 patients from 2014 to 2018 were recovered. Assume participants were randomized to intervention and control, respectively; 3 had events from each arm and 3 dropouts from each arm. From the figure, dotted transparent figures denote LTFU participants, whereas red dotted figures denote LTFU participants being assumed with event. The plausible assumptions being made toward the LTFU was based on relative event incidence and a formula detailed in Data S6. The number of events were assumed based on the reported formula with incidence ranging from 1 to 3. Calculations of how many trials' relative risks lost significance after making the assumptions were run subsequently. Ann of Intern Med, Annals of Internal Medicine; Eur Heart J, European Heart Journal; JACC, Journal of the American College of Cardiology; LTFU, loss to follow-up; NEJM, New England Journal of Medicine.

Percentage of Trials Losing Significance
For the 4 common assumptions in which all 91 trials were included, the percentages of trials that lost significance were 4% (no participants lost to follow-up had the event), 11% (all participants lost to follow-up had the event), 3% (best-case scenario), and 33% (worstcase scenario).
Considering the relative event incidence analysis method, Table 4 shows the percentage of eligible trials that lost significance across a range of assumptions for the event incidence among intervention and control arms ( Figure 5). The percentage varied from 2% to 16%. Figure 6 shows an inverse-proportion relationship of the trials losing significance with the percentage of LTFU under the best and worst assumptions made by the relative event incidence analysis method.
Results of the prespecified sensitivity analysis on the subspecialties are reported (interventional cardiology versus others) in the online Data Supplement. There was a significant difference in the proportion of trials losing significance between interventional cardiology and other subspecialties (difference, 4.35% [95% CI, 0.295%-8.41%]; P=0.0369; Figure S1 and Table S2).

DISCUSSION
We found considerable variation in the extent and reporting of LTFU data in contemporary cardiovascular clinical trials. We observed that certain characteristics of clinical trials-notably, inadequate or unclear allocation concealment, length of follow-up, sample size, and type of intervention-were associated with increased odds of LTFU. Importantly, the primary result Figure 6. Distribution of trials by LTFU proportion under the best and worst plausible assumptions made by using the relative event incidence for the control and intervention arms. Distribution of trials losing statistical significance stratified by the percentage of LTFU of the individual trial under the best and worst assumptions made by the more plausible relative event incidence method. An inverse-proportion relationship is shown in the graph, where there is higher number of trials losing significance in trials with lower proportions of LTFU. LTFU indicates loss to follow-up.

Issues That Should Be Noted Guidance
Inadequate or unclear allocation concealment If allocation concealment forms part of the trial design, then effective approaches to achieve allocation concealment include using a matched placebo (visually identical to the active treatment); central randomization (performed at a site remote from the trial's location); sequentially numbered, sealed, opaque envelopes 3 Large sample size and long follow up duration LTFU increases with larger trial sample size, hence investigators should be aware and mitigate the number of LTFU for increase in sample size and 1-y increase in duration Reporting of LTFU Investigators should strive to reduce the number of LTFU. A CONSORT diagram should be included for readers. When LTFU has occurred, baseline characteristics, and extent of follow up duration before exclusion should be reported in the manuscript or supplement. The implications of LTFU should also be discussed in the manuscript. Time of dropout can be noted on a supplement or in the result paragraph or on the CONSORT diagram for readers to know the extent of follow up before dropout CONSORT indicates Consolidated Standards of Reporting Trials; LTFU, loss to follow-up. in 1 of 6 trials might change if reasonable assumptions were made about the end point in patients with LTFU. The inverse-proportion relationship noted in Figure 6 suggests that the impact of a small proportion of LTFU might be overlooked by investigators. More than one third of trials did not achieve effective blinding among either the participants or the site investigators. This finding is important because ineffective blinding is associated with overestimation of true treatment effects. 135 Allocation concealment was inadequate or unclear in more than half of the trials. Conversely, an intention-to-treat analysis was used in 75% of trials, which minimizes the effects of attrition bias. 3 Just over half of the trials included an explicit statement about LTFU, and >70% of the trials included a CONSORT flow diagram. Notably, baseline demographics on the LTFU participants were limited. Authors (93%) commonly omitted discussion of the potential impact or reasons for LTFU. We suggest that information on participants with LTFU should be included by authors in an appendix or in a defined column in a table of the trial participants' characteristics (Table 5). Inverse probability weighting can be a helpful way of handling LTFU participants' data, but it is not used in any of the included trials. Most trials did not impute data for LTFU participants. We noted a significant association between inadequate or unclear allocation concealment and increased odds of LTFU. This could be explained by less stringently implemented processes in trials with inadequate or unclear allocation concealment, including suboptimal measures for following up participants.

Strengths and Weaknesses of the Study
Our study has several strengths. First, the forms for screening of the trials and related data collection were established before the start of the data collection process. In addition, the calibration exercise was completed upfront as a preparatory step intended to increase accuracy for the screening and data collection. Second, a range of assumptions was made for the participants with LTFU and explored the potential effect of LTFU on the estimate of the effect of the intervention, including whether or not the trial met statistical significance on its primary outcome and the change in the relative risk ratio and number of outcome events. The effect is focused on cardiovascular trials. Our analysis depends on the accuracy and clarity of the included reports. Generalizability is also an issue. We focused our analysis on 8 journals' publications during a 5year period (2014-2018). A wider inclusion strategy with more journals (with lower impact factors) and trials with a nonsignificant primary outcome result might have returned different results. Our findings might underestimate the true effect of LTFU in the effect estimate if a wider range of RCTs were included. Our review included trials with binary data only because of the design of the review analysis, which might further weaken the generalizability of the results. Time of dropout can be a factor influencing the LTFU effect because early dropouts can influence the result to a larger extent than late dropouts. However, exact time of dropout is not noted in the reports, and we are unable to stratify the effect.

Implications
Investigators and sponsors should strive to reduce the number of participants with LTFU. The higher the LTFU, the more uncertainty increases around the treatment effect estimate and the potential for a false result. In the unfortunate event that LTFU happened, its impact can be estimated using the worst assumption (Data S7). As for the reporting of LTFU, editors may consider requiring authors to provide a fully informative and transparent report on participant LTFU including the inclusion and exclusion criteria of patients, which is in line with CONSORT guidelines. Specifically, investigators should provide information on participants with LTFU including their baseline characteristics, reasons for LTFU, and duration of follow-up before exclusion and then compared with those who completed follow-up. This information could be published as an appendix. Implications of LTFU should be discussed when LTFU has occurred (Table 5). This review provides estimates of the probability that the primary analysis of cardiovascular trials could lose statistical significance when LTFU events are taken into account by making appropriate estimate of event incidence. Although the 4 less plausible but commonly used assumptions may not eventuate, they can be taken as the upper limit of change in trial significance. Early LTFU has a more influential effect on the analysis than late LTFU near the overall study duration, which highlighted the need for investigators to stratify LTFU by follow-up extent. Future studies can look at the extent of change in treatment effect in relation to the LTFU proportion and event number and the effect of partial and full LTFU defined as difference in the extent of followup before exclusion. The influence of dropout time on LTFU effect can be explored for assessing the possibility of systemic inclusion of patients accounting for early dropouts.

CONCLUSIONS
Almost 1 in 6 (16%) cardiovascular randomized trials published in leading journals may have a change in the primary outcome if plausible assumptions are made about differential event rates of participants lost to follow-up. There is scope for improvement arising from LTFU in randomized trials in cardiovascular medicine. Bias minimization through mitigation of participants lost to follow-up offers the opportunity to enhance the value of randomized trials.

Acknowledgments
Author contributions: Fong, Ford, and Berry were responsible for study conception and design. Fong and Ford acquired the data. Fong, Jüni, and da Costa analyzed the data. Fong drafted the manuscript. All authors critically revised the manuscript and agreed on the final version.

Sources of Funding
The study was funded by British Heart Foundation (PG/17/2532884; RE/13/5/30177; RE/18/6134217). The funder had no role in the study design, the writing of manuscript, or the decision to submit this article or future manuscripts for publication.

Disclosures
Berry is employed by the University of Glasgow which holds consultancy and research agreements for his work with companies that have commercial interests in the diagnosis and treatment of angina. The companies include Abbott Vascular, Astra Zeneca, Boehringer Ingelheim, GSK, HeartFlow, Menarini, Novartis, and Siemens Healthcare. Jüni serves as unpaid member of the steering group of trials funded by Astra Zeneca, Biotronik, Biosensors, St. Jude Medical and The Medicines Company, has received research grants to the institution from Astra Zeneca, Biotronik, Biosensors International, Eli Lilly and The Medicines Company, and honoraria to the institution for participation in advisory boards from Amgen, but has not received personal payments by any pharmaceutical company or device manufacturer. The remaining authors have no disclosures to report.

. Extent of loss to follow-up
The extent of LTFU was estimated by calculating the percentage of LTFU in each trial from each arm (intervention and control). Then, median and interquartile range of the percentages across trials were obtained. A ratio of the total number of participants identified as LTFU to the number of primary outcome events was calculated for each trial (the "lost to follow-up to events ratio"). Median and standard deviation of this ratio was also calculated across the trials.

c. Potential impact of loss to follow-up
The potential impact of LTFU is evaluated by proposing assumptions about the outcomes of participants Although the above assumptions are widely used in multiple literatures, some experts have countered they are not plausible and have suggested a novel method for estimating effects of LTFU. 17 Akl et al evaluated more plausible assumptions that the incidence of events among participants lost to follow-up is higher by a specific ratio relative to the observed event incidence among participants followed up. 17 They defined RILTFU/FU as the event incidence among those lost to follow-up relative to the event incidence among those followed up and made plausible assumptions towards the outcome of LTFU participants (see data S6). LTFU refers to "lost to follow-up" and FU refers to "followed up". A range of plausible RILTFU/FU values (1,1.5,2,3) was used in both the intervention group and the control group.
3 is the upper limit, which was previously determined by consensus. 17

Data S6 -Illustrations of the assumptions being considered in FLUKE with examples
Examples based on the following theoretical trial: -Randomization: 100 to intervention while 100 to control group -Loss to follow up: 20 in the intervention group while 10 in the control group -Events: 15 in the intervention group while 20 in the control group • No events experienced by any lost to follow-up participants =0% • Events experienced by all the lost to follow-up participants =17% • Events only experienced by the LTFU in control group while no events experienced by the LTFU in treatment group (best case scenario) =0% • Events only experienced by the LTFU in treatment group while no events experienced by the LTFU in control group (worst case scenario) =22% Among the 68 trials from other cardiology field, percentage which results would lose significance under different assumptions: • No events experienced by any lost to follow-up participants =6% • Events experienced by all the lost to follow-up participants =9% • Events only experienced by the LTFU in control group while no events experienced by the LTFU in treatment group (best case scenario) =4% • Events only experienced by the LTFU in treatment group while no events experienced by the LTFU in control group (worst case scenario) =37%