Good Laboratory Practice: Preventing Introduction of Bias at the Bench

Nearly 10 years after the first Stroke Therapy Academic Industry Roundtable participants established guidelines intended to support the translation of neuroprotective efficacy from bench to bedside,¹ there is still no clinically effective neuroprotective drug for stroke. One interpretation of this observation is that measures outlined in STAIR I have failed to deliver the promised improvements in drug development. However, a dispassionate analysis of data presented over the last 10 years suggests that the “STAIR hypothesis”—that improvements in animal experimental design will lead to improvements in translational efficiency—has yet to be adequately tested. Adhering to standards of conducting and reporting of experiments in order to reduce the confounding effects of bias and ensure adequate statistical power, as outlined below, will increase the confidence with which we can assess new data and maximize our chances of developing effective therapies.

The original STAIR proposal was that by paying due attention to experimental bias, to the breadth of physiological variables known to influence stroke outcome in patients, and by testing therapies in a range of model systems which might more faithfully reproduce the key facets of stroke pathophysiology, we would be able to translate what appeared to be clear evidence of neuroprotective efficacy in animals to the more heterogeneous circumstances of human stroke. Although we believe strongly that failure to adequately consider variables such as age, comorbidity, physiological status, and timing of drug administration contribute to the disparity between the results of animal models and clinical trials, they have been reviewed elsewhere^1,2 and are not the subject of this article.

Analyses of data supporting the efficacy of various neuroprotective strategies^3–5 have revealed that although many researchers adhere closely to the ethos of these guidelines, as a community we do not. A simple checklist derived from the STAIR guidelines to provide an overview of the range of data available for 1026 candidate therapies⁴ revealed that only a handful came close to meeting the STAIR guidelines. A higher score against this checklist was accompanied by a marked reduction in effect size. This later trend could be seen clearly even within the data for individual drugs.⁶ Moreover, studies which reported measures to avoid bias such as random allocation to treatment group, blinded induction of ischemia or the blinded assessment of outcome,^7,8 gave a markedly lower estimate of efficacy. Despite this, there has been some evidence of improvement in study quality, and the performance of animal stroke studies is substantially better than that for most other models of neurological disease.⁵ And yet, the majority of investigators still do not report whether or not they took measures to avoid bias.

Systematic reviews and meta-analyses of data from animal stroke studies suggest that these studies may be substantially distorted by experimental bias. Taken together, publications supporting the efficacy of NXY-059 include randomized data with allocation concealment and blinded outcome assessment, but most individual publications do not report these measures. Analyses of those data suggests that at least half of the reported 44% improvement in outcome could be attributed to experimental bias, specifically a failure to randomize the allocation to experimental group, a failure to conceal treatment group allocation from the surgeon or a failure to blind the assessment of outcome.⁸ Similar observations have been made of the hypothermia literature, where nonrandomized studies and studies without blinded outcome assessment appear to give a relative overstatement of efficacy of 27% and 19%, respectively.⁷ Despite the widely recognized importance of these aspects of study design, analyses conducted by the CAMARADES group suggest that only 36% of studies reported random allocation to treatment group, only 11% report allocation concealment and only 29% reported the blinded assessment of outcome.⁵

A related issue is the number of animals used in experiments. The probability of detecting a difference of a given size between groups is related to the number of animals in each group, the size of the difference and the variability in the outcome measure used. However, only 3% of studies identified in systematic reviews reported using a sample size calculation.⁵ Importantly, if sample size calculations are based on falsely large estimates of effect size, studies will not be powered to detect real differences between treatment and control groups. Indeed, post hoc analysis suggests that most experimental stroke studies have only a 1 in 3 chance of detecting a 20% difference in outcome.

These problems are not unique to the preclinical study of stroke. Clinical stroke trials have had problems with inadequate sample size⁹ and have also failed to report whether they took measures to avoid bias.¹⁰ Indeed Cochrane’s observation that “when humans have to make observations there is always the possibility of bias”¹¹ was a lynchpin of the CONSORT (Consolidated Standards of Reporting Trials) initiative to improve the reporting, design, conduct, analysis, and interpretation of randomized controlled trials to inform decision-making in health care.^11,12 This initiative led to substantial improvements in the reporting and conduct of clinical trials.¹³

On the basis of the available evidence, it would now seem reasonable to suggest that preclinical testing in animal models of stroke, and indeed other models of disease, should adopt similar standards to ensure that decision-making is based on high quality unbiased data.^5,14 Adoption of such standards would have the added benefit of reducing wasteful use of financial and animal resources.

In general, studies should only be considered for publication if their Methods section includes a description of how they have addressed the standards below, or if authors make a cogent argument for why these standards are not relevant to their work. For these components of an article, citation of methods described in previous publications is not considered sufficient. These requirements should not preclude publication of important observational, pilot or hypothesis-generating data, but the conclusions of such studies should reflect their preliminary nature.