Stroke Quality Metrics: Systematic Reviews of the Relationships to Patient-Centered Outcomes and Impact of Public Reporting

Despite substantial progress since the release of the Institute of Medicine's Crossing the Quality Chasm report,¹ significant work remains in terms of defining, measuring, and improving the quality of health care in the United States. We recently reported on the methodological development and implementation of stroke performance measures (or quality metrics) in the United States.² Endorsed stroke quality metrics play central roles in ongoing stroke quality improvement (QI) programs, including the Primary Stroke Center Certification Program,³ the Paul Coverdell National Acute Stroke Registry,⁴ and the Get With The Guidelines (GWTG)-Stroke Program.⁵ Although randomized studies have yet to be conducted, there is good evidence from observational studies of QI programs that the systematic collection, evaluation, and feedback of stroke quality metric data can result in rapid improvements in the quality of care.⁶ For example, analysis of >322 000 ischemic stroke or transient ischemic attack patients discharged from 790 hospitals participating in the GWTG-Stroke program over a 5-year period found clinically meaningful improvements in the quality of stroke care that were independent of underlying secular trends.⁶ Subsequent analyses of the GWTG-Stroke data strongly suggest that the increased compliance with stroke quality metrics resulted from an increase in the proportion of eligible subjects treated (ie, improvements in the quality of care), rather than an increase in the number of patients excluded from measure denominators.⁷

However, currently, several important challenges remain, limiting the potential for stroke quality metrics to be fully incorporated into broader programmatic and regulatory policies.² First, the degree to which greater compliance with stroke quality metrics results in direct improvements in patient-centered outcomes remains to be proven for stroke. Second, although public reporting of performance data from health care providers resonates with many, several concerns remain, including the validity of the measures reported, whether public reporting directly improves quality of care,⁸ or whether it leads to inaccurate characterizations of hospitals and providers.⁹,¹⁰

The objectives of this article are to conduct systematic reviews of observational studies examining the relationship between stroke quality metrics and patient-centered outcomes and public reporting of stroke quality metrics and QI activity, quality of care, or patient-centered outcomes.

Studies included in the systematic review of stroke quality metrics and patient outcomes were limited to observational studies that evaluated the relationship between compliance with ≥2 quality metrics and patient-centered outcomes in a broad population of acute stroke admissions. Additional criteria excluded studies that were conducted in a single hospital setting or that were limited to specific subpopulations of age, race, sex, or stroke subtype. Eligible quality metrics were not limited to those advocated by any specific accrediting body, government institution, or nonprofit corporation. Patient-centered outcomes included measures of mortality, disability, physical function, complications, quality of life, and patient satisfaction.¹¹ The literature search was conducted using MEDLINE and EMBASE and included studies published before December 31, 2010. We conducted several searches by first combining the term “stroke” with the following key words or phrases: “quality of care,” “performance measure*,” “quality improvement,” “quality metric,” “quality measure,” and “process* of care.” These searches were then combined with the following outcomes-related terms: “outcome*,” “complications,” “mortality,” and “patient satisfaction.” We supplemented the search by screening the bibliographies of the articles that underwent full review. The search was limited to articles with an English language abstract.

For the relevancy screen, 2 authors (CP, MR) independently reviewed each title and abstract. Articles identified as potentially relevant by either reviewer underwent a full independent review to determine if the article met inclusion criteria. Final determination of study eligibility was performed by consensus. We excluded studies that evaluated the efficacy of stroke units because an extensive evidence base including meta-analyses of randomized and nonrandomized studies already exists.^12,13 Similarly, we excluded studies that examined the impact of stroke center certification¹⁴ if they did not directly evaluate individual stroke metrics. Because of heterogeneity of the studies, we were unable to assess study quality.

A similar approach was used to identify studies that addressed the public reporting of stroke quality metrics and the relationship to QI activity, quality of care, or patient-centered outcomes. Adapting terms used in a previous systematic review,¹⁵ we conducted several searches by combining the term “stroke” with the following key words or phrases: “public report,” “public reporting,” “provider profile,” “report card,” “public information,” “consumer information,” “patient information,” “consumer report,” “public performance report,” and “public performance.” We also supplemented the search by screening the bibliographies of the articles that underwent full review.

Our systematic review confirmed that most stroke quality metrics primarily document process-centered aspects of acute hospital-based stroke care, and that most were founded on evidence-based guidelines or accreditation needs. However, we found that there is a limited evidence base addressing the relationships between quality metric compliance and stroke-related outcomes. Of the 14 studies included in this review, 9 reported mostly positive associations between stroke quality metric compliance and stroke outcomes. Although we found only a relatively modest evidence base supporting the relationship between stroke quality metric compliance and improved patient outcomes, it could be argued that this evidence base is stronger than that present in the literature for other cardiovascular diseases (ie, acute myocardial infarction and heart failure), in which the number of significant associations indicating a positive relationship between higher quality of care and better patient outcomes is surprisingly limited.^32–35

In addition to the limited number of studies identified, the authors of these studies have cautioned against overinterpreting their results because of the fundamental methodological challenges associated with determining causal relationships between stroke metrics and patient outcomes.³⁶ There are at least 5 distinct methodological challenges associated with determining these causal associations. First, statistical risk adjustment methods for stroke outcomes are not sufficiently developed to satisfactorily account for differences in patient populations (eg, age, stroke severity, comorbidities) across health care providers and settings.^28,37,38 Unfortunately, there is wide variability in the number and scope of patient-level variables included in risk adjustment models and the statistical methods used to develop and validate them. In addition, many organizations generating data for public reporting use proprietary and undisclosed methods, highlighting the need for greater oversight and transparency.³⁹ Second, analyses of facility-level data are often hampered by small sample sizes, both in the number of subjects who are eligible for a given metric, as well as the number of subjects experiencing a given outcome, eg, mortality or complications. The resulting imprecision translates to findings that vary substantially across sites or over time because of random error alone.

Third, confounding by indication, which describes the phenomenon whereby medical treatment is directly influenced by the patient's prognosis,⁴⁰ is a fundamental limitation of observational studies examining the association between quality metrics and patient outcomes. Confounding by indication can result in paradoxical associations between quality of care and outcomes. For example, patients with severe stroke have inherently poor prognosis but may be treated more aggressively and therefore may receive higher-quality care. Similarly, better care may be delivered to patients who have a higher burden of risk factors or comorbidities,⁴¹ whereas, conversely, patients with mild disease or few comorbidities are not treated as aggressively. The net impact of these different confounding mechanisms may be impossible to discern.

A fourth challenge is determining the relevant time window for measuring outcomes relative to when care was provided. Outcomes measured soon after the provision of care (ie, ambulatory status at discharge) are unlikely to accurately reflect the quality or effectiveness of the care provided. Equally, outcomes measured many months after care was provided are unlikely to be a valid reflection of the care provided. The American Hospital Association/American College of Cardiology task force on performance measures has recommended that stroke outcomes be measured 1 month after hospital discharge.⁴² Finally, although in-hospital mortality is readily measured, it is not a good reflection of the quality of care provided.^43–45 In-hospital mortality varies according to length of stay, is impacted by hospital policies related to hospice use, and may be influenced by patient/family preferences. Patients with care directives that limit the aggressiveness of medical care after a disabling stroke who receive compassionate or hospice care resulting in death are treated in many mortality models as having experienced a poor or adverse outcome rather than high-quality care. Given these limitations, there is a critical need to expand stroke mortality measures to include longer time periods (ie, 30 days, 90 days, and 1 year) and to incorporate other patient-centered outcomes, such as care preferences, functional outcomes, and quality of life.

We found only 2 studies that directly addressed the relationship between public reporting of stroke quality metrics and either QI activity, quality of care, or patient-orientated stroke outcomes. Several other reports have been motivated by the concerns raised about public reporting of stroke metrics,^24,28 but these did not meet our inclusion criteria for the review. Public reporting of performance data has been a primary objective of many consumer advocates, insurance industry, and government agencies, including the Centers for Medicare and Medicaid Services.⁴⁶ The central premise is that public reporting will lead to higher quality of care because providers will undertake efforts to improve care and consumers will choose plans or hospitals that provide better care.⁴⁷ Solid evidence is still lacking.

Outside of stroke research, results of studies examining the effects of public reporting on quality or patient outcomes have been quite variable. There have been at least 2 systematic reviews; in 2000, Marshall et al¹⁵ conducted a review of the impact of public disclosure of performance data but could only find 7 United States-based peer-reviewed studies. They concluded that hospitals were most responsive to the publication of performance data, but found limited evidence that this was associated with improved health outcomes.¹⁵ In 2008, a systematic review of 11 studies also found that public reporting stimulated QI activity in hospitals but did not affect hospital selection by patients.⁴⁸ The authors found mixed evidence for an association between public reporting and better patient outcomes, but commented that the included studies were mostly descriptive in nature and had limited strength of evidence.⁴⁸ In more recent studies, a follow-up analysis of the Hospital Compare program,⁴⁶ a public reporting initiative by the Centers for Medicare and Medicaid Services, found that public reporting was associated with increased performance, especially in hospitals that were low performers at baseline.⁴⁹ This increased performance was shown to positively affect by reducing mortality, length of stay, and readmission rates for acute myocardial infarction patients.⁴⁹ A recent randomized study of 86 Ontario hospitals evaluated the effect of public reporting on quality of care provided to acute myocardial infarction and heart failure patients.⁵⁰ Public report cards documenting baseline care performance were either released early for 44 randomly selected hospitals or delayed (by 18 months) for 42 remaining hospitals. Hospitals randomized to early data release were significantly more likely to start QI initiatives compared to hospitals receiving delayed feedback (73.2% versus 46.7% and P=0.003 for acute myocardial infarction care; 61.0% versus 50.0% and P=0.04 for heart failure care); however, they did not show any significant improvements in composite process-of-care measures.

Whether mandatory public reporting of stroke quality metric data at the hospital or physician level will induce greater or more widespread quality gains compared to what has already been achieved through the current voluntary, confidential reporting approach remains to be determined.⁶ It is noteworthy that, as evidenced by the widespread participation in national-level stroke QI registries,⁵¹ there is currently widespread tacit support for using quality metrics for QI purposes that does not rely on public reporting.

Our report has several potential limitations. First, because we excluded research specific to evaluation of stroke units or hospital stroke center certification, our conclusions do not generalize to these 2 areas. Second, it is likely that our search strategies missed relevant articles, and that our exclusion criteria may have prevented consideration of all pertinent reports. Third, the limited number of peer-reviewed reports on public reporting of stroke metrics likely reflects the impact of publication bias in that relevant research conducted on this topic has, for whatever reason, failed to become peer-reviewed literature. Fourth, heterogeneity of study designs precluded any formal quality scoring assessment efforts or the generation of quantitative summary estimates of effect. Finally, we note that some articles included in our review were based on the same underling data source (ie, the Veteran's Health Administration^18–20 and the Danish Indicator Project^22,23). Although some of these data may represent duplicate cases, each study met our inclusion criteria because it reported on different outcome measures.

In summary, although much work has been performed regarding validating individual stroke quality metrics and increasing data reliability, there remain challenging methodological difficulties—including inadequate risk adjustment, small sample sizes, confounding by indication, appropriate time window, and use of in-hospital mortality—to proving a causal relationship between quality metrics and stroke outcomes. Because of the fundamental problems in measuring and interpreting outcome-based measures from observational studies, several authors have cautioned against using outcome measures to gauge quality of care, advocating instead that process measures should remain the primary method of comparing quality across hospitals and providers.²⁶,³⁶,³⁷,⁴²,⁴⁴ The challenges identified in this review should be addressed before quality metrics are incorporated widely into programmatic and regulatory policies, particularly those associated with provider reimbursement and institutional accreditation. There is great need for the generation of valid evidence demonstrating the relationship between stroke quality metric compliance and patient-centered outcomes, as well as the public reporting of these metrics, so that the current considerable investments undertaken to improve stroke care can be sustained.