Good Clinical Practice Guidance and Pragmatic Clinical Trials
Abstract
Randomized, clinical trials are commonly regarded as the highest level of evidence to support clinical decisions. Good Clinical Practice guidelines have been constructed to provide an ethical and scientific quality standard for trials that involve human subjects in a manner aligned with the Declaration of Helsinki. Originally designed to provide a unified standard of trial data to support submission to regulatory authorities, the principles may also be applied to other studies of human subjects. Although the application of Good Clinical Practice principles generally led to improvements in the quality and consistency of trial operations, these principles have also contributed to increasing trial complexity and costs. Alternatively, the growing availability of electronic health record data has facilitated the possibility for streamlined pragmatic clinical trials. The central tenets of Good Clinical Practice and pragmatic clinical trials represent potential tensions in trial design (stringent quality and highly efficient operations). In the present article, we highlight potential areas of discordance between Good Clinical Practice guidelines and the principles of pragmatic clinical trials and suggest strategies to streamline study conduct in an ethical manner to optimally perform clinical trials in the electronic age.
Introduction
Health care decision makers need evidence-based medicine to support clinical and health policy choices,1 and randomized, clinical trials are the highest level of evidence to support these decisions.2,3 Good Clinical Practice (GCP) guidelines were developed to provide an ethical and scientific quality standard for investigators, sponsors, monitors, and institutional review boards throughout each stage of clinical trials.4 These guidelines were initially designed to harmonize conduct for clinical trials intending to submit data to regulatory authorities. GCP principles are commonly applied to contemporary clinical investigations of human subjects with the intent of supporting the safety and well-being of study participants. GCP serves “as a roadmap of responsibilities” for those involved in research and can improve the quality and consistency of trial operations.5
Editorial, see p 823
In contradistinction, it has been suggested that some of these guidelines if inflexibly applied may result in challenges.6 For instance, some GCP processes can lead to markedly increased trial complexity, duration, and costs without substantially improving the quality of these trials, their ability to correctly answer clinical questions or support the safety of human subjects.7,8 Furthermore, sponsor interpretation of GCP may complicate trial conduct via implementation of regulatory and monitoring approaches that increase the workload and dissatisfaction of site staff and research monitors as well as study participants.
More recently, the growing widespread availability of electronic health record (EHR) data in community practice has led to the potential to use such data to streamline trials and conduct pragmatic clinical trials (PCTs).9,10 EHR-based PCTs represent 1 contemporary strategy to improve the efficiency of clinical trials, reduce costs, and support more real-world study conduct. Although recent EHR-empowered trial designs offer remarkable opportunities, there is a potential tension between certain central tenets outlined in GCP guidelines and the core principles of PCTs. In the present article, we highlight potential areas of discordance between GCP guidelines and the principles of PCTs and suggest strategies to balance these perspectives to optimally perform clinical trials in the electronic age.
The Increasing Cost and Complexity of Contemporary Trials
Many clinical trials in the 1980s and 1990s were characterized by relatively streamlined protocols with assessment of a few hard clinical outcomes (eg, mortality), modest financial support for sites, and few regulatory hurdles. For instance, the International Studies of Infarct Survival (ISIS) studies assessing acute myocardial infarction (MI) therapies used a 1-page case report form without site monitoring, end point adjudication, or site payments.11,12 In this era, the integrity of the trial results was supported by randomization, large sample sizes, and unbiased outcome assessment. However, a number of competing forces including financial incentives and perceived conflicts of interest led to increased trial bureaucracy.12 In response, GCP guidelines were developed to ensure patient safety, prevent and detect fraud, and ensure the validity of trial findings. However, these guidelines also led to increased trial complexity and cost. Challenges with participant recruitment and retention, declining funding, and poor engagement of recruiting clinicians placed further strain on the research infrastructure.13 With increased documentation and regulatory requirements, and a strained research structure, as well, clinical trial enrollment shifted from the United States and Western Europe to other world regions. Factors promoting the globalization of clinical research and the potential adverse impact of these trends were recently outlined.14 The current paradigm shift in clinical trials requires a reappraisal of GCP guidelines and their applicability to contemporary research.
Several studies from the Tufts Center for the Study of Drug Development have evaluated the cost and complexity associated with study conduct in the contemporary GCP environment. In 2011, Getz et al15 reported temporal changes in protocol design complexity and study staff burden in >8300 clinical trial protocols from 2000 to 2007 (Table 1). The typical phase III protocol in the 2000 to 2003 period had an average of 20 unique procedures in comparison with 28 unique procedures conducted 5 times in 2004 to 2007. In cardiovascular trials, there was a ≈50% increase in total procedures and 30% increase in total work burden in the comparison of these periods. A separate analysis found that for phase III studies ≈25% of procedures supported regulatory requirements and noncore data with an average direct cost of ≈$2 million (≈20% of the total).16 The authors conservatively estimated that the total direct costs for these procedures in active phase II and III studies is ≈$4 billion annually. Intensive study-specific testing not only influences the burden on site staff, but also reduces patient participation in clinical trials.17
| 2000–2003 | 2004–2007 | Change, % | |
|---|---|---|---|
| Unique procedures | 20.0 | 28.4 | 42.0 |
| Total procedures | 93.6 | 147.5 | 57.6 |
| Total work burden, units | 27.0 | 43.1 | 59.7 |
| Median CRF pages | 55 | 180 | 227.3 |
A recent analysis also assessed temporal changes in the size, duration, and enrollment rates for cardiovascular trials published between 2001 and 2012 in select high-impact journals.18 Comparing the 2001 to 2003 period with 2009 to 2012, trials involved more patients (median from 400 to 500) and sites (from 20 to 22) and enrollment rate decreased from 1.2 to 0.9 patients/site per month. Importantly, low enrollment rates may influence event rates and the validity of trial results.19,20 Median trial duration (2.1 years) did not significantly change over time.
Taken together, these temporal changes are attributable to a number of factors including advances in medical therapy that require larger numbers to demonstrate a net clinical benefit, and an overall globalization of trials and regulatory challenges, as well.14,21 Regardless, these overall trends in trial complexity, work burden, cost, and enrollment challenges suggest the need for a reappraisal of the current trial environment.
Central Tenets of GCP: Advantages and Criticisms
The initial intention of GCP criteria was to ensure the safety and rights of participants in trials and the reliability of trial data to support the safety of future patients.4 In brief, the guidelines detail the responsibilities, procedures, and recording that are necessary for appropriate trial conduct by investigators, study staff, sponsors, and institutional review boards. In general, these guidelines have led to improved quality of clinical trial conduct and reporting. For instance, GCP stipulates that trials are conducted in compliance with the institutional review board–approved protocol with appropriate adverse event monitoring and reporting. These consistent expectations supported a reduction in site/investigator misconduct, enhanced the protection of patients, and improved data quality and standardization across trials for reporting to regulatory bodies.
Importantly, the development and implementation of GCP guidance were related to and complementary to other documents focusing on the ethics of human subject research; these concepts are covered in documents including the Declaration of Helsinki,22 the Belmont Report,23 and the Code of Federal Regulations, Protection of Human Subjects.24 We refer the reader to previous summary statements on these documents.25,26 In brief, the Declaration of Helsinki was developed by the World Medical Association’s Committee on Medical Ethics in 1964 to serve as a guide for physicians engaged in clinical research involving human subjects.25 Basic principles include that studies be conducted by qualified persons with the health, interests, privacy, and integrity of the patient as the first consideration with careful assessment of “predictable risk and foreseeable benefit.” The Belmont Report was subsequently developed in 1978 by the US National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research to articulate the fundamental ethical principles underlying human subjects research: beneficence, justice, and respect for persons.26 The US Department of Health and Human Services has also developed a specific Code of Federal Regulations detailing basic policies for protection of human subjects (Title 45, Part 46).24 This document outlines policies related to informed consent and protection of special populations and applies to all research involving human participants that is subject to regulation by any US federal department or agency. Thus, GCP guidance not only includes details on scientific quality standards, but also stems from the principles outlined in documents involving the ethics of human subject research.
However, not all of the downstream effects of GCP implementation have been evidence based or positive. In some circumstances, there is an inappropriate emphasis on reporting (eg, progress reports, safety reporting, final study reports), monitoring, auditing, and essential documents as a result of GCP guidelines. Other reviews6,27 have previously critiqued the challenges of GCP guidance. In brief, the guidance is derived from informal consensus rather than evidence-based data. The document’s writing process did not include academic researchers well versed in trial methodology. In fact, the documents lack details on authorship and references are not included. As a result, key details are missing from the documents. One example that is commonly highlighted is that there is no discussion on adequate allocation concealment in randomized trials. Allocation concealment refers to the technique used to implement the sequence of randomization in a manner that keeps clinicians and participants blinded to treatment assignment.28 The lack of appropriate concealment has been shown to introduce bias and significantly influence trial quality.29 Moreover, despite the significant changes in the clinical research environment over the past decade, the documents have not been updated since 1996.
Implementation of GCP guidelines has also resulted in intensive site monitoring, which is costly and time consuming and has unproven benefit.6 Key GCP considerations include clinical monitoring and data audits to confirm that trial data are “verifiable from source documents.” As discussed by Reith et al,6 “they focus to an inappropriate extent on ensuring the completeness and accuracy of each piece of data that is recorded, even though minor errors occurring with similar frequency in the treatment groups should not materially affect the findings.” Whereas, in many circumstances, site monitoring has become less intensive over time,30 some ongoing trials continue to require substantial on-site verification of documents detailed in GCP guidelines. Major goals include the detection of fraud and the inaccurate transcription of data. Although these goals are important, the tendency to emphasize minutia may direct focus away from critical aspects of trial conduct, such as appropriate patient enrollment and retention, study intervention, and outcome assessment. There is an emphasis on essential documents unrelated to research validity (eg, updating curriculum vitae, signature sheets, drug-storage records, temperature logs) rather than activities designed to directly improve research quality and enhance scientific validity (eg, minimizing lost to follow-up). Although the intention of GCP is to protect and promote patients’ rights and safety while increasing the overall quality of clinical trials, the interpretation and implementation of GCP guidelines has been far from ideal.
PCT Background, Study Characteristics, and Limitations
Although contemporary trials seem to be becoming more complex, and costly, the emergence of EHR and registry data offer the possibility of novel PCT designs whose form and function are much more similar to the streamlined clinical trials of the 1970s and 1980s.31,32 The key characteristics of PCTs have been previously reviewed.33,34 In brief, PCTs focus on whether the intervention is effective under usual conditions or in a real-world setting rather than under controlled ideal circumstances. In general, PCTs have broad entry criteria to enroll a diverse study population across heterogeneous practice settings to enhance the generalizability of study results, a shortcoming of traditional, highly selected clinical trials.35 A major aim of PCTs is to simplify eligibility criteria, screening, and overall study conduct to improve trial efficiency.
Clinically relevant alternative interventions are compared such that results are readily applicable to usual care. Certain types of interventions may be more appropriate in a PCT than in a conventional trial. For instance, blinding and placebo-controlled comparators may be cost prohibitive in the setting of a large-scale PCT aligned with usual care. To balance concerns related to the scientific rigor or validity of a trial conducted without blinding, additional design considerations may be necessary such as the use of objective end points (eg, all-cause mortality) and use of an end point–capture strategy that is systematic for all patients. Similarly, the strategy to address monitoring procedures is also not a simple one-size-fits-all approach and may require a tailored approach to match the research goals in a manner that optimizes ethical and scientific quality.
Study conduct is incorporated into routine clinical practice rather than through study-specific visits, reducing the burden and inconvenience to participants. The adherence of practitioners and patients is assessed in an unobtrusive manner (if at all). End points are clinically meaningful with assessment in a manner that is consistent with usual care (ie, site-reported) with less reliance on central adjudication. Thus, although the development of so-called pragmatic trials appears to be a relatively recent paradigm, in many respects, the design of PCTs has its roots in earlier clinical trials as exemplified by the ISIS and Global Utilization of Streptokinase and Tissue Plasminogen Activator for Occluded Coronary Arteries (GUSTO)36 studies. More recently, to help trialists design studies and characterize the study’s position on the spectrum from explanatory to pragmatic, the pragmatic-explanatory continuum indicator summary (PRECIS) tool was developed34 and updated.37 This tool provides a framework to characterize the different components of a clinical trial (eg, entry criteria, follow-up schedule) to promote study designs that are consistent with the intended degree of pragmatism.
The design considerations of PCTs represent 2 sides of a coin with respect to strengths and limitations. Trials that enroll a broad patient population have a more heterogeneous study population than trials with strict entry criteria. On 1 hand, this may improve the generalizability of study results. However, when neutral results are observed in a heterogeneous population, it may be uncertain whether the intervention does not work or whether positive results would have been observed under optimal conditions (ie, effectiveness versus efficacy design). Effectiveness studies may leverage a pragmatic methodology to assess interventions under usual circumstances within a broader population targeting formulary approval and real-world comparative effectiveness. In contrast, efficacy studies are routinely designed with an explanatory methodology that seeks to answer whether an intervention works under optimal circumstances. There is usually a strictly defined patient population and the results may be targeting regulatory approval. The US Food and Drug Administration requires placebo-controlled studies for pharmaceutical approval (unless superiority over another marketed product is targeted) and has historically required efficacy rather than effectiveness data.33 As a result, most previous and ongoing PCTs compared clinically available therapeutics (ie, phase IV studies) rather than novel therapeutics requiring regulatory review. Other potential concerns with PCTs include limitations of real-world safety reporting, reduced patient retention and the potential for reduced adherence and data acquisition outside the context of conventional monitoring, and study-specific visits/procedures.
Progress in Pragmatism: Registry-Based Studies and Cluster Randomized Trials
Several strategies that have been used in contemporary clinical trials to improve pragmatism include leveraging additional data sources such as registries and incorporating cluster designs to streamline trial conduct. The Thrombus Aspiration in Myocardial Infarction (TASTE) trial assessed whether thrombus aspiration during ST-segment–elevation MI reduced mortality in a multicenter trial with enrollment of patients and end point acquisition from national registries in Sweden.38 Study of Access Site for Enhancement of Percutaneous Coronary Intervention for Women (SAFE-PCI for Women) was a randomized trial comparing radial versus femoral arterial access in women undergoing percutaneous coronary intervention (PCI).39 The study embedded the randomized trial into the existing infrastructure of the National Cardiovascular Data Registry CathPCI Registry through the National Institutes of Health’s National Cardiovascular Research Infrastructure. Similarly, the Treatment with Adenosine Diphosphate Receptor Inhibitors-Longitudinal Assessment of Treatment Patterns and Events After Acute Coronary Syndrome (TRANSLATE-ACS) trial was a longitudinal observational study of MI patients managed with PCI.40 The trial design built on the PCI registry platform and incorporated a systematic telephone interview follow-up process, a cluster-randomized substudy to investigate the utility of platelet inhibition testing, and an assessment of the dissemination of site-specific, quality-of-care data benchmarked to peer performance. Additional clinical trials have incorporated streamlined cluster designs as in the Post-Myocardial Infarction Free Rx and Economic Evaluation (Post-MI FREEE) trial, which assessed the clinical impact of reducing cost sharing for cardiac medications following MI.41 These clinical trial designs represent incremental progress toward improved pragmatism in comparison with the so-called explanatory or conventional trials that do not incorporate the streamlined processes proposed in PCTs.
EHR-Facilitated PCTs
The growth of EHR data across health systems has generated enthusiasm for EHR-facilitated PCTs. Importantly, although the EHR is a rich source of clinical data, it is specifically designed to support clinical care and reimbursement. The assumption that EHR data are fit for use in high-quality clinical research has not been rigorously evaluated to date. Several ongoing investigations will assess the fitness of EHR data to facilitate efficient, reliable, and cost-effective clinical research. For instance, the Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-term Effectiveness (ADAPTABLE trial) is one of the first chronic intervention trials that leverages a PCT design in comparison with more conventional randomized, clinical trials. ADAPTABLE is assessing lower-dose or higher-dose aspirin in patients with heart disease through the Patient Centered Outcomes Research Network (PCORnet) infrastructure.42 PCORnet is a coordinated network of Clinical Data Research Networks representing health system collaborations and Patient-Powered Research Networks of patients/stakeholders with representation in all 50 states and coverage of >25 million Americans. The trial is embedded within usual care with minimal entry criteria, electronic patient-directed consent, and data collection that incorporates data that have been standardized to a common data model, Medicare claims, and patient-reported outcomes. The trial will recruit 20 000 patients, and, by leveraging EHR data with reduced burden on patients, clinicians, and practices, it is expected to cost less than conventional trials.
An example of a previous interventional PCT that leveraged EHR data for outcome acquisition was a real-world, randomized, open-label trial of antidepressants.43 In addition, the Time to Reduce Mortality in End-Stage Renal Disease (TiME) Trial is an ongoing cluster-randomized National Institutes of Health collaboratory trial that is assessing the implications of hemodialysis duration on clinical outcomes and quality of life through a pragmatic design that leverages EHR data and collaboration between academic investigators and industry (clinicaltrials.gov identifier: NCT02019225). Moreover, the BPMedTime Trial is an National Heart, Lung, and Blood Institute–funded randomized pragmatic trial designed to evaluate the safety and efficacy of nighttime dosing of antihypertensive medications that will leverage the EHR for recruitment and data acquisition at 2 collaborating health systems with follow-up for 36 to 42 months (National Institutes of Health Project Reporter: 1UH2AT007784-01).
Despite these recent initial steps toward streamlining clinical trials, EHR-facilitated PCTs face significant challenges. Our understanding of how EHR-based platforms might increase the efficiency of data collection, outcome surveillance, and confirmation in an integrated manner is in a nascent state. For instance, evidence from the West of Scotland Coronary Prevention Study suggests that cardiovascular end points can be ascertained from routinely recorded EHR-type data, but classification was imperfect.44 Specifically, fatal end points ascertained via routine mechanisms (ie, source documentation, case report form completion, adjudication) showed excellent matching with register data (97%). However, nonfatal events such as MI and stroke matched ≈80% of the time. These discrepancies were attributable to factors including duplicate events, disagreement between site-reported and adjudicated events, events outside the catchment area, miscoding, and linkage errors. Importantly, the authors demonstrated that the observed risk reduction for the primary end point differed by using the record-linked data in comparison with the original trial results (39% risk reduction [24–51] versus 29% [15–44]), yet the qualitative conclusions were similar regardless of the methodology.
Similarly, a population-based study in Sweden comparing heart failure diagnoses in the hospital register with adjudication via chart review found that ≈80% of register cases were classified as definite heart failure with echocardiographic data increasing this to ≈90%.45 Lower validity was observed when comparing general medicine clinics versus cardiology clinics (86% versus 91%), but the assessment of heart failure as the primary diagnosis increased validity to 95%. Taken together, these examples suggest that event classification varies depending on the specific end point but is ≥80% for commonly assessed cardiovascular end points. Importantly, the methodology for classification using both structure and unstructured EHR data is a rapidly growing area of research and will likely continue to improve over time.
EHRs are inherently heterogeneous, and common data elements are defined in different ways.46,47 The complexity of the healthcare system impacts the EHR as well. Patients receive care from multiple providers48 and in multiple healthcare systems,49 so complete information does not exist at a single location. Thus, although significant advances have been made in recent years with respect to EHR-facilitated trials, many challenges remain.
Comparison of Conventional Trials and EHR-Facilitated Trials
EHR-facilitated trials may offer financial advantages over more conventional explanatory trials in the contemporary GCP environment. A recent conventional trial of >14 000 diabetic patients enrolled at 660 sites from 2008 to 2012 with follow-up through 2015 cost nearly $250 million with monitoring constituting > $56 million (23%). In comparison, the ADAPTABLE trial that leverages EHR data to target enrollment of 20 000 patients over a shorter enrollment period is estimated to cost ≈$14 to 18 million with reduced costs for trial management and monitoring and increased costs for informatics.42
Tension Between GCP and PCTs
Areas of potential tension exist between the guidance provided by GCP and the key characteristics of PCTs. From a patient enrollment perspective, GCP guidelines indicate the need for screening and written informed consent procedures by a qualified study team member, whereas PCTs emphasize streamlined identification of study participants via electronic mechanisms and consideration of mass enrolling. In addition, there is growing interest in pragmatic trials using patient-directed consent and electronic consent50 rather than more traditional coordinator/investigator–facilitated consent. Additional details related to study personnel responsibilities are detailed throughout the GCP guideline document. For instance, GCP indicates that qualified physician investigators should be responsible for all trial-related medical decisions and patient follow-up related to the use of the investigational product, and medical care for comorbid conditions, as well. The responsibilities outlined in the GCP guidelines may appear at odds with PCTs that emphasize inclusion of practitioners with a range of research experience, flexibility of study intervention in routine care, and minimal trial-specific training. Documentation, monitoring, and reporting guidelines are extensively detailed in the GCP document, whereas PCTs leverage routinely collected data with monitoring that is risk based (ie, scalable depending on need). A formal process for clinical event classification is invoked via the language in GCP, whereas PCTs support streamlined event capture and minimal (if any) formal adjudication. Streamlined safety reporting32 including large-scale surveillance of safety data51 is central to PCTs, whereas GCP details on-site monitoring with auditing and source document verification.
Harmonization of GCP and PCTs
Despite the potential for tension between GCP guidelines and PCT characteristics, innovative approaches to clinical trials can be harmonized with these historic trial guidelines. Although prior trials designed to support US Food and Drug Administration approval were routinely conventional trials with explanatory methodology, we propose that future trials across the spectrum of product development should consider incorporating elements of pragmatism as able. Ultimately, each trial design should be constructed in an individualized manner that is fit for purpose. Rather than a 1-size-fits-all approach to trial design, different trials may incorporate various degrees of operational simplicity while leveraging available data, PCT concepts, and logical implementation of GCP. Table 2 provides an overview of potential solutions to address GCP guidance in the context of PCTs. For example, a strategy of simplifying the informed consent process can be conducted in a manner that reduces the burden on patients and investigators, while supporting the rights and safety of patients in a manner consistent with GCP recommendations. Previous studies have demonstrated that, when patients do not read informed consent documents, they are nearly 3 times more likely to decline trial participation in comparison with those patients who read the document.52 Therefore, a simplified consent form in language accessible to the layperson may be designed in collaboration with patient partners to balance the need to adequately inform the patient of critical study elements and convey any risks to the patient. These consent forms should focus on the presentation at the eighth-grade level with typical oral reading rates of no more than 150 words per minute to ensure comprehension53 and should also implement comprehension questions. Similarly, the historic perspective that medical contact for those involved in all trials needs to be orchestrated by qualified study investigators through study-specific visits may represent an overinterpretation of the GCP guidelines that can be adapted to the PCT model. A potential advantage of clinical trials that are aligned with routine care is reduction in the Hawthorne effect that may be seen with additional study visits and patient monitoring. The Hawthorne effect involves an alteration in the behavior of research participants as a consequence of the awareness of study participation, which may bias trial results.54
| Domain | Potential Solutions |
|---|---|
| Patient enrollment and consent | • Patient identification via electronic health record search with real-time point-of-care triggers to alert clinician of potential eligibility. Such strategies may support patient autonomy and justice with improved access to available trials while also reducing the burden of trial participation on clinicians. |
| • Simplified informed consent text and procedures to reduce burden on patients/families and investigators while supporting the rights, safety, and well-being of patients. | |
| ○ Enhanced efforts to better inform patients regarding expectations, responsibilities, and rights for trial participants in contemporary clinical practice that capitalize on multimedia approaches that are individualized to patients (eg, video platforms for trial overview and consent) | |
| Study intervention and medical care | • Incorporation of intervention as part of routine care by usual care provider with input/mentorship from a qualified trial physician |
| ○ Protocol orders (eg, tests and procedures) and visit schedules are incorporated into EHR orders and documentation | |
| High-quality data | • Implementation of risk-based site-monitoring strategies |
| • Central statistical monitoring of sites | |
| • Streamlined event adjudication as the preferred option, if appropriate, with formal clinical events committee as needed | |
| Study personnel | • A real-world clinical trial team that includes members with varied training and experience with appropriate support from a qualified physician and trial champion |
| Study visits, follow-up, and documentation | • Incorporate existing electronic data from routine follow-up into the study documentation, and registry data and national registry data, as well (as able) |
| ○ Follow-up via ongoing clinical registries, Centers for Medicare and Medicaid Services claims, and direct patient contact (eg, email and telephone contact) | |
| Monitoring and adherence measures | • Focus on consent procedures, randomization integrity, safety, and complete follow-up |
| Reporting and safety | • Streamline reporting with an emphasis on generating safety data that are similar to those reviewed by a Data Safety Monitoring Board and leveraging safety procedures used in routine clinical care (eg, MedWatch) rather than the historic perspectives of industry safety groups |
Focused and abbreviated study-specific and general research training of usual care providers may be appropriate to support a PCT model that leverages streamlined participation of a real-world clinical trial team with support and mentorship from a trial physician. Furthermore, for some PCTs that are aligned with standard-of-care procedures, minimal (if any) additional research training or study-specific training may be needed for usual care providers. With the ascertainment of baseline and outcomes data from electronic sources including the EHR, registries, and national registers, the responsibility and burden of research participation for enrolling sites may be reduced and the requirements for provider training may not apply or may be reduced. A balanced approach for monitoring and safety reporting would incorporate large-scale surveillance of safety data51 as able, with more conventional reporting of specific adverse events of interest. Importantly, some clinical trials (eg, a cardiovascular outcomes study for a novel agent) may be best conducted with a more traditional trial design that incorporates streamlined operations, end points, and data acquisition, as appropriate, to reduce cost and improve efficiency.
At the present time, there are several ongoing efforts that may improve harmonization of GCP perspectives with contemporary clinical trials. The National Heart Lung and Blood Institute also has keen interest in reforming trial design and conduct.55,56
TransCelerate BioPharma is a nonprofit entity that includes biopharmaceutical companies, regulatory bodies, and academicians with a mission of collaborating across the research community “to identify, prioritize, design and facilitate the implementation of solutions to drive efficient, effective and high-quality delivery of new medicines.”57 The consortium has focused on improving the quality and efficiency of clinical trials via incremental advancements in the following areas: risk-based monitoring,58 site qualification and training that meets benchmarked minimum GCP criteria, industry-wide clinical data standards to support research data exchange and patient safety, and the development of a shared investigator platform to exchange data and protocols to facilitate trial development. Additional initiatives of TransCelerate include the creation of common clinical trial protocol templates and a global investigator registry to streamline trial conduct and optimize trial efficiency with supporting appropriate trial conduct and patient safety.
The Clinical Trials Transformation Initiative (CTTI) was cofounded in 2007 by Duke University and the US Food and Drug Administration to identify and promote clinical trial practices that prioritize quality and efficiency.59 CTTI’s membership includes academic research organizations and representatives from industry and government, and patients and investigators, as well. The group has generated data on clinical trial conduct to provide recommendations for improvement on topics such as informed consent, patient recruitment, and institutional review board conduct. Several of the specific areas of advancement that have been the focus of CTTI to date include the development of a Quality by Design document that includes evidence-based recommendations for improving trial quality60 and collaboration with the US Food and Drug Administration–established Mini-Sentinel program to facilitate future randomized trials that leverage the distributed database model.61
Additional think-tanks including representation from academia, industry, and regulatory bodies have extended these discussions on improving clinical trial conduct to topics including data safety–monitoring board processes,62 postmarketing evaluations,63 and reducing racial and sex disparities in clinical trials.64
Future Directions
Despite the potential tension between GCP guidance and PCT methodology, we have highlighted strategies to help harmonize and individualize the guidance as applied to PCTs. These considerations may inform future trial design and conduct. In addition, these areas of tension suggest the need to revise and update the historic GCP guidelines to improve relevance to the contemporary research environment. GCP reform is necessary not only for the implementation of PCTs, but also for the improvement of the efficiency of conventional trials. The inclusion of academic trialists, patient partners, and evidence-based data in these revisions will be necessary. We suggest the possibility of a reduced emphasis on monitoring, auditing, and essential documents. Rather, we favor shifting the focus to stream-lined and real-world enrollment, study conduct, and reporting to ensure internal and external validity of trial results. The guidance could benefit from changing the guidelines to more specifically cover Good Clinical Trial Practice in the contemporary research environment.6 In brief, the emphasis should be on making sure that the right patient (ie, satisfies entry criteria with adequate consent) receives the appropriate intervention (ie, correct randomization, blinding, and treatment assignment) with adequate assessment of outcomes (ie, complete, correct, and timely event ascertainment). With the appropriate engagement of patients, clinicians, researchers, policy makers, and regulators, these issues can be clarified to improve the clinical research enterprise while maintaining high standards of protection of the rights, safety, and well-being of study participants. TransCelerate, CTTI, and Mini-Sentinel represent ongoing collaborative efforts that may help to harmonize GCP principles with the design of PCTs. GCP does not necessarily preclude the conduct of PCTs, but rather requires alignment between the different trial stakeholders, particularly sponsors. A central theme is the alignment of industry subgroups, including those from compliance, regulatory, and safety groups, with those of investigators designing PCTs. Implementation of GCP is dependent in large part on differential interpretation by these parties. Improved partnerships between regulators, industry representatives, trialists, and patients on the interpretation of GCP is long overdue, and the era of EHR-facilitated PCTs may represent the ideal time for a reappraisal and redesign of study conduct.
Sources of Funding
Dr Mentz receives research support from the
Disclosures
Dr Mentz receives research support from Amgen, AstraZeneca, Bristol-Myers Squibb, GlaxoSmithKline, Gilead, Novartis, Otsuka, and ResMed; honoraria from HeartWare, Janssen, Luitpold Pharmaceuticals, Novartis, ResMed, and Thoratec; and has served on an advisory board for Luitpold Pharmaceuticals, Inc. Dr Hernandez reported consulting for Amgen, AstraZeneca, Bayer, BMS, Corthera, Cytokinetics, Gilead, GSK, Janssen, Johnson & Johnson, Merck, Pluristem Therapeutics and Sensible; research support from Amylin, Bristol-Myers Squibb, Johnson & Johnson, GSK, Merck, Portola. Dr Curtis receives funding from Boston Scientific, GlaxoSmithKline, Gilead, NCQA, and Novartis. Dr Peterson has received institutional grant support from the American College of Cardiology, the American Heart Association, Eli Lilly, and Janssen; and consulting fees (including Continuing Medical Education) from Merck & Co., Boehringer Ingelheim, Bristol-Myers Squibb, Pfizer, Genentech, Janssen, and Sanofi. Dr O’Brien receives research support from the NIH, PCORI, Bristol-Myers Squibb, GlaxoSmithKline, Janssen Scientific, and Boehringer-Ingelheim and has served on an advisory board for Portola Pharmaceuticals. The other authors report no conflicts.
Footnotes
References
- 1.
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS . Evidence based medicine: what it is and what it isn’t.BMJ. 1996; 312:71–72.CrossrefMedlineGoogle Scholar - 2. ACCF/AHA Task Force on Practice Guidelines. Methodology Manual and Policies From the ACCF/AHA Task Force on Practice Guidelines. 2010. http://assets.cardiosource.com/Methodology_Manual_for_ACC_AHA_Writing_Committees.pdf Accessed May 12, 2015.Google Scholar
- 3.
Collins R, MacMahon S . Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials.Lancet. 2001; 357:373–380. doi: 10.1016/S0140-6736(00)03651-5.CrossrefMedlineGoogle Scholar - 4. International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. ICH Harmonized Tripartite Guideline. Guideline for Good Clinical Practice E6(R1). 1996.http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R1_Guideline.pdf Accessed May 12, 2015.Google Scholar
- 5.
Devine S, Dagher RN, Weiss KD, Santana VM . Good clinical practice and the conduct of clinical studies in pediatric oncology.Pediatr Clin North Am. 2008; 55:187–209, xi. doi: 10.1016/j.pcl.2007.10.008.CrossrefMedlineGoogle Scholar - 6.
Reith C, Landray M, Devereaux PJ, Bosch J, Granger CB, Baigent C, Califf RM, Collins R, Yusuf S . Randomized clinical trials–removing unnecessary obstacles.N Engl J Med. 2013; 369:1061–1065. doi: 10.1056/NEJMsb1300760.CrossrefMedlineGoogle Scholar - 7.
Califf RM . Clinical trials bureaucracy: unintended consequences of well-intentioned policy.Clin Trials. 2006; 3:496–502. doi: 10.1177/1740774506073173.CrossrefMedlineGoogle Scholar - 8.
McMahon AD, Conway DI, Macdonald TM, McInnes GT . The unintended consequences of clinical trials regulations.PLoS Med. 2009; 3:e1000131. doi: 10.1371/journal.pmed.1000131.CrossrefMedlineGoogle Scholar - 9.
Frankovich J, Longhurst CA, Sutherland SM . Evidence-based medicine in the EMR era.N Engl J Med. 2011; 365:1758–1759. doi: 10.1056/NEJMp1108726.CrossrefMedlineGoogle Scholar - 10.
Collins FS, Hudson KL, Briggs JP, Lauer MS . PCORnet: turning a dream into reality.J Am Med Inform Assoc. 2014; 21:576–577. doi: 10.1136/amiajnl-2014-002864.CrossrefMedlineGoogle Scholar - 11. Randomised trial of intravenous atenolol among 16 027 cases of suspected acute myocardial infarction: ISIS-1. First International Study of Infarct Survival Collaborative Group.Lancet. 1986; 2:57–66.MedlineGoogle Scholar
- 12.
Yusuf S . Randomized clinical trials: slow death by a thousand unnecessary policies?CMAJ. 2004; 171:889–892; discussion 892. doi: 10.1503/cmaj.1040884.CrossrefMedlineGoogle Scholar - 13.
Butler J, Fonarow GC, O’Connor C, Adams K, Bonow RO, Cody RJ, Collins SP, Dunnmon P, Dinh W, Fiuzat M, Georgiopoulou VV, Grant S, Kim SY, Kupfer S, Lefkowitz M, Mentz RJ, Misselwitz F, Pitt B, Roessig L, Schelbert E, Shah M, Solomon S, Stockbridge N, Yancy C, Gheorghiade M . Improving cardiovascular clinical trials conduct in the United States: recommendation from clinicians, researchers, sponsors, and regulators.Am Heart J. 2015; 169:305–314. doi: 10.1016/j.ahj.2014.12.001.CrossrefMedlineGoogle Scholar - 14.
Glickman SW, McHutchison JG, Peterson ED, Cairns CB, Harrington RA, Califf RM, Schulman KA . Ethical and scientific implications of the globalization of clinical research.N Engl J Med. 2009; 360:816–823. doi: 10.1056/NEJMsb0803929.CrossrefMedlineGoogle Scholar - 15.
Getz KA, Campo RA, Kaitin KI . Variability in protocol design complexity by phase and therapeutic area.Drug Inform J. 2011; 45:413–420.CrossrefGoogle Scholar - 16.
Getz KA, Stergiopoulos S, Marlborough M, Whitehill J, Curran M, Kaitin KI . Quantifying the magnitude and cost of collecting extraneous protocol data.Am J Ther. 2015; 22:117–124. doi: 10.1097/MJT.0b013e31826fc4aa.CrossrefMedlineGoogle Scholar - 17.
Martin SS, Ou FS, Newby LK, Sutton V, Adams P, Felker GM, Wang TY . Patient- and trial-specific barriers to participation in cardiovascular randomized clinical trials.J Am Coll Cardiol. 2013; 61:762–769. doi: 10.1016/j.jacc.2012.10.046.CrossrefMedlineGoogle Scholar - 18.
Butler J, Tahhan AS, Georgiopoulou VV, Kelkar A, Lee M, Khan B, Peterson E, Fonarow GC, Kalogeropoulos AP, Gheorghiade M . Trends in characteristics of cardiovascular clinical trials 2001–2012.Am Heart J. 2015; 170:263–272.e2.CrossrefMedlineGoogle Scholar - 19.
Butler J, Subacius H, Vaduganathan M, Fonarow GC, Ambrosy AP, Konstam MA, Maggioni A, Mentz RJ, Swedberg K, Zannad F, Gheorghiade M ; EVEREST Investigators. Relationship between clinical trial site enrollment with participant characteristics, protocol completion, and outcomes: insights from the EVEREST (Efficacy of Vasopressin Antagonism in Heart Failure: Outcome Study with Tolvaptan) trial.J Am Coll Cardiol. 2013; 61:571–579. doi: 10.1016/j.jacc.2012.10.025.CrossrefMedlineGoogle Scholar - 20.
Lovato LC, Hill K, Hertert S, Hunninghake DB, Probstfield JL . Recruitment for controlled clinical trials: literature summary and annotated bibliography.Control Clin Trials. 1997; 18:328–352.CrossrefMedlineGoogle Scholar - 21.
Bailey W, Cruickshank C, Sharma N . Make your move: taking clinical trials to the best location. https://www.atkearney.com/health/ideas-insights/article/-/asset_publisher/LCcgOeS4t85g/content/make-your-move/10192. Accessed October 7, 2015.Google Scholar - 22. World Medical Association. Declaration of Helsinki serves as guide to physicians.JAMA. 1964; 189:33–34.CrossrefGoogle Scholar
- 23. U.S. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical Principles and Guidelines for the Conduct of Research Involving Human Subjects. Washington, DC: US Department of Health, Education, and Welfare, 1978.Google Scholar
- 24. US Department of Health and Human Services. Code of Federal Regulations. Title 5, Part B. Public Welfare: Protection of Human Subjects. http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html. Accessed October 7, 2015.Google Scholar
- 25. World Medical Association Declaration of Helsinki. Recommendations guiding physicians in biomedical research involving human subjects.JAMA. 1997; 277:925–926.CrossrefMedlineGoogle Scholar
- 26.
Sims JM . A brief review of the Belmont report.Dimens Crit Care Nurs. 2010; 29:173–174. doi: 10.1097/DCC.0b013e3181de9ec5.CrossrefMedlineGoogle Scholar - 27.
Grimes DA, Hubacher D, Nanda K, Schulz KF, Moher D, Altman DG . The Good Clinical Practice guideline: a bronze standard for clinical research.Lancet. 2005; 366:172–174. doi: 10.1016/S0140-6736(05)66875-4.CrossrefMedlineGoogle Scholar - 28.
Schulz KF, Grimes DA . Allocation concealment in randomised trials: defending against deciphering.Lancet. 2002; 359:614–618. doi: 10.1016/S0140-6736(02)07750-4.CrossrefMedlineGoogle Scholar - 29.
Mills EJ, Ayers D, Chou R, Thorlund K . Are current standards of reporting quality for clinical trials sufficient in addressing important sources of bias?Contemp Clin Trials. 2015; 45(pt A):2–7. doi: 10.1016/j.cct.2015.07.019.CrossrefMedlineGoogle Scholar - 30. US Department of Health and Human Services. Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations - A Risk-Based Approach to Monitoring. www.fda.gov/downloads/Drugs/Guidances/UCM269919.pdf Accessed August 17, 2015.Google Scholar
- 31.
Fordyce CB, Roe MT, Ahmad T, Libby P, Borer JS, Hiatt WR, Bristow MR, Packer M, Wasserman SM, Braunstein N, Pitt B, DeMets DL, Cooper-Arnold K, Armstrong PW, Berkowitz SD, Scott R, Prats J, Galis ZS, Stockbridge N, Peterson ED, Califf RM . Cardiovascular drug development: is it dead or just hibernating?J Am Coll Cardiol. 2015; 65:1567–1582. doi: 10.1016/j.jacc.2015.03.016.CrossrefMedlineGoogle Scholar - 32.
Roe MT, Mahaffey KW, Ezekowitz JA, Alexander JH, Goodman SG, Hernandez A, Temple T, Berdan L, Califf RM, Harrington RA, Peterson ED, Armstrong PW . The future of cardiovascular clinical research in North America and beyond-addressing challenges and leveraging opportunities through unique academic and grassroots collaborations.Am Heart J. 2015; 169:743–750. doi: 10.1016/j.ahj.2015.03.002.CrossrefMedlineGoogle Scholar - 33.
Tunis SR, Stryer DB, Clancy CM . Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.JAMA. 2003; 290:1624–1632. doi: 10.1001/jama.290.12.1624.CrossrefMedlineGoogle Scholar - 34.
Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K . A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.J Clin Epidemiol. 2009; 62:464–475. doi: 10.1016/j.jclinepi.2008.12.011.CrossrefMedlineGoogle Scholar - 35.
Rothwell PM . External validity of randomised controlled trials: “to whom do the results of this trial apply?”.Lancet. 2005; 365:82–93. doi: 10.1016/S0140-6736(04)17670-8.CrossrefMedlineGoogle Scholar - 36. An international randomized trial comparing four thrombolytic strategies for acute myocardial infarction. The GUSTO investigators.N Engl J Med. 1993; 329:673–82.CrossrefMedlineGoogle Scholar
- 37.
Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M . The PRECIS-2 tool: designing trials that are fit for purpose.BMJ. 2015; 350:h2147.CrossrefMedlineGoogle Scholar - 38.
Fröbert O, Lagerqvist B, Olivecrona GK, Omerovic E, Gudnason T, Maeng M, Aasa M, Angerås O, Calais F, Danielewicz M, Erlinge D, Hellsten L, Jensen U, Johansson AC, Kåregren A, Nilsson J, Robertson L, Sandhall L, Sjögren I, Ostlund O, Harnek J, James SK ; TASTE Trial. Thrombus aspiration during ST-segment elevation myocardial infarction.N Engl J Med. 2013; 369:1587–1597. doi: 10.1056/NEJMoa1308789.CrossrefMedlineGoogle Scholar - 39.
Hess CN, Rao SV, Kong DF, Aberle LH, Anstrom KJ, Gibson CM, Gilchrist IC, Jacobs AK, Jolly SS, Mehran R, Messenger JC, Newby LK, Waksman R, Krucoff MW . Embedding a randomized clinical trial into an ongoing registry infrastructure: unique opportunities for efficiency in design of the Study of Access site For Enhancement of Percutaneous Coronary Intervention for Women (SAFE-PCI for Women).Am Heart J. 2013; 166:421–428. doi: 10.1016/j.ahj.2013.06.013.CrossrefMedlineGoogle Scholar - 40.
Chin CT, Wang TY, Anstrom KJ, Zhu B, Maa JF, Messenger JC, Ryan KA, Davidson-Ray L, Zettler M, Effron MB, Mark DB, Peterson ED . Treatment with adenosine diphosphate receptor inhibitors-longitudinal assessment of treatment patterns and events after acute coronary syndrome (TRANSLATE-ACS) study design: expanding the paradigm of longitudinal observational research.Am Heart J. 2011; 162:844–851. doi: 10.1016/j.ahj.2011.08.021.CrossrefMedlineGoogle Scholar - 41.
Choudhry NK, Brennan T, Toscano M, Spettell C, Glynn RJ, Rubino M, Schneeweiss S, Brookhart AM, Fernandes J, Mathew S, Christiansen B, Antman EM, Avorn J, Shrank WH . Rationale and design of the Post-MI FREEE trial: a randomized evaluation of first-dollar drug coverage for post-myocardial infarction secondary preventive therapies.Am Heart J. 2008; 156:31–36. doi: 10.1016/j.ahj.2008.03.021.CrossrefMedlineGoogle Scholar - 42.
Hernandez AF, Fleurence RL, Rothman RL . The ADAPTABLE Trial and PCORnet: shining light on a new research paradigm.Ann Intern Med. 2015; 163:635–636. doi: 10.7326/M15-1460.CrossrefMedlineGoogle Scholar - 43.
Simon GE, VonKorff M, Heiligenstein JH, Revicki DA, Grothaus L, Katon W, Wagner EH . Initial antidepressant choice in primary care. Effectiveness and cost of fluoxetine vs tricyclic antidepressants.JAMA. 1996; 275:1897–1902.CrossrefMedlineGoogle Scholar - 44.
Barry SJ, Dinnett E, Kean S, Gaw A, Ford I . Are routinely collected NHS administrative records suitable for endpoint identification in clinical trials? Evidence from the West of Scotland Coronary Prevention Study.PLoS One. 2013; 8:e75379. doi: 10.1371/journal.pone.0075379.CrossrefMedlineGoogle Scholar - 45.
Ingelsson E, Arnlöv J, Sundström J, Lind L . The validity of a diagnosis of heart failure in a hospital discharge register.Eur J Heart Fail. 2005; 7:787–791. doi: 10.1016/j.ejheart.2004.12.007.CrossrefMedlineGoogle Scholar - 46.
Bayley KB, Belnap T, Savitz L, Masica AL, Shah N, Fleming NS . Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied.Med Care. 2013; 51(8 suppl 3):S80–S86. doi: 10.1097/MLR.0b013e31829b1d48.CrossrefMedlineGoogle Scholar - 47.
Botsis T, Hartvigsen G, Chen F, Weng C . Secondary Use of EHR: data quality issues and informatics opportunities.AMIA Jt Summits Transl Sci Proc. 2010; 2010:1–5.MedlineGoogle Scholar - 48.
Pham HH, Schrag D, O’Malley AS, Wu B, Bach PB . Care patterns in Medicare and their implications for pay for performance.N Engl J Med. 2007; 356:1130–1139. doi: 10.1056/NEJMsa063979.CrossrefMedlineGoogle Scholar - 49.
Bourgeois FC, Olson KL, Mandl KD . Patients treated at multiple acute health care facilities: quantifying information fragmentation.Arch Intern Med. 2010; 170:1989–1995. doi: 10.1001/archinternmed.2010.439.CrossrefMedlineGoogle Scholar - 50.
Geller NL, Kim DY, Tian X . Smart technology in lung disease clinical trials.Chest. 2016; 149:22–26. doi: 10.1378/chest.15-1314.CrossrefMedlineGoogle Scholar - 51.
Psaty BM, Breckenridge AM . Mini-sentinel and regulatory science–big data rendered fit and functional.N Engl J Med. 2014; 370:2165–2167. doi: 10.1056/NEJMp1401664.CrossrefMedlineGoogle Scholar - 52.
Sen Biswas M, Newby LK, Bastian LA, Peterson ED, Sugarman J . Who refuses enrollment in cardiac clinical trials?Clin Trials. 2007; 4:258–263. doi: 10.1177/1740774507079434.CrossrefMedlineGoogle Scholar - 53.
Hasbrouck J, Tindal GA . Oral reading fluency norms: a valuable assessment tool for reading teachers.The Reading Teacher. 2006; 59:636–644.CrossrefGoogle Scholar - 54.
McCambridge J, Witton J, Elbourne DR . Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects.J Clin Epidemiol. 2014; 67:267–277. doi: 10.1016/j.jclinepi.2013.08.015.CrossrefMedlineGoogle Scholar - 55.
Lauer MS, Kiley JP, Mockrin SC, Mensah GA, Hoots WK, Patel Y, Cook NL, Patterson AP, Gibbons GH . National Heart, Lung, and Blood Institute (NHLBI) strategic visioning: setting an agenda together for the NHLBI of 2025.Circulation. 2015; 131:1106–1109. doi: 10.1161/CIRCULATIONAHA.115.015712.LinkGoogle Scholar - 56.
Lauer MS, D’Agostino RB . The randomized registry trial–the next disruptive technology in clinical research?N Engl J Med. 2013; 369:1579–1581. doi: 10.1056/NEJMp1310102.CrossrefMedlineGoogle Scholar - 57.
Gill D . Re-inventing clinical trials through TransCelerate.Nat Rev Drug Discov. 2014; 13:787–788. doi: 10.1038/nrd4437.CrossrefMedlineGoogle Scholar - 58. TransCelerate. Risk-Based Monitoring. http://www.transceleratebiopharmainc.com/assets/risk-based-monitoring/ on Accessed August 12, 2015.Google Scholar
- 59.
Tenaerts P, Madre L, Archdeacon P, Califf RM . The Clinical Trials Transformation Initiative: innovation through collaboration.Nat Rev Drug Discov. 2014; 13:797–798. doi: 10.1038/nrd4442.CrossrefMedlineGoogle Scholar - 60. CTTI. Quality by Design. http://www.ctti-clinicaltrials.org/files/QbD_QRM/QbDQRM-PrinciplesDocument-DeviceWorkshop-2014-01-16.pdf Accessed August 12, 2015.Google Scholar
- 61. Mini-Sentinel and Clinical Trials Transformation Initiative. Developing Approaches to Conducting Randomized Trials using the Mini-Sentinel Distributed Database. http://www.mini-sentinel.org/work_products/Statistical_Methods/Mini-Sentinel_Methods_CTTI_Developing-Approaches-to-Conducting-Randomized-Trials-Using-MSDD.pdf Accessed August 12, 2015.Google Scholar
- 62.
Hess CN, Roe MT, Gibson CM, Temple RJ, Pencina MJ, Zarin DA, Anstrom KJ, Alexander JH, Sherman RE, Fiedorek FT, Mahaffey KW, Lee KL, Chow SC, Armstrong PW, Califf RM . Independent data monitoring committees: preparing a path for the future.Am Heart J. 2014; 168:135–41.e1. doi: 10.1016/j.ahj.2014.05.003.CrossrefMedlineGoogle Scholar - 63.
Rao SV, Califf RM, Kramer JM, Peterson ED, Gross TP, Pepine CJ, Williams DO, Donohoe D, Waksman R, Mehran R, Krucoff MW . Postmarket evaluation of breakthrough technologies.Am Heart J. 2008; 156:201–208. doi: 10.1016/j.ahj.2008.01.036.CrossrefMedlineGoogle Scholar - 64.
Berger JS, Bairey-Merz CN, Redberg RF, Douglas PS . Improving the quality of care for women with cardiovascular disease: report of a DCRI Think Tank, March 8 to 9, 2007.Am Heart J. 2008; 156:816–25, 825.e1.CrossrefMedlineGoogle Scholar


