Using the National Institutes of Health Stroke Scale

Introduction

The National Institutes of Health Stroke Scale (NIHSS) is the most widely used deficit rating scale in modern neurology: over 500 000 healthcare professionals have been certified to administer it using a web-based platform. Every clinical trial in vascular neurology—prevention, acute treatment, recovery—requires a severity assessment, and the NIHSS became the gold standard for stroke severity rating after the first successful trial in acute stroke therapy, the NINDS r-tPA (National Institute of Neurological Disorders and Stroke recombinant tissue-type plasminogen activator) for Acute Stroke Trial (the Trial).¹ As part of the Trial, detailed and rigorous training/certification procedures were created for the NIHSS that facilitate wider use of the scale outside of research.²

Today, payers and regulators demand reportable data on patient outcomes, and such outcomes must be adjusted for baseline severity: the NIHSS has become the de facto metric for regulatory compliance. The Joint Commission, as part of its certification program for Primary Stroke Centers, now requires an NIHSS score within 12 hours of admission for all stroke patients; this assessment is to be done by a certified examiner.^3,4 Federal agencies also require outcomes adjusted for baseline stroke severity—using the NIHSS.⁵ Despite widening regulatory requirements, considerable problems may arise in using the NIHSS in clinical practice because the scale was designed for research purposes.⁶ Given that the scale was not designed for such widespread—and determinative—application, anyone using (or mandating use of) the NIHSS must understand its development history, clinimetric properties, and its proper bedside administration.

History/Development

During the late 1980s, several stroke-deficit rating scales were in use.^7–10 For use in a National Institutes of Health–sponsored trial of naloxone for acute stroke, investigators combined scales that had been developed at the University of Cincinnati, Canadian neurological scale, the Edinburgh-2 coma scale, and the Oxbury initial severity scale.¹¹ Greater scores correlated with larger infarctions.¹² This Cincinnati/Naloxone version of the NIHSS served the intended purpose in the Naloxone trial.¹³ An intermediate version was used in the Pilot r-tPA for Acute Stroke Trial,¹⁴ but when designing the NINDS r-tPA for Acute Stroke Trial, significant modifications were made to facilitate using the NIHSS in a larger clinical trial.¹⁵ The version used today is this final iteration of the NIHSS, and it differs in important ways from the Cincinnati/Naloxone NIHSS (Table 1). A modified version contains fewer, more reliable items.¹⁶

The final (r-tPA) version of the NIHSS was validated against infarct volumes.¹⁷ Several scale items require intact language function, so the NIHSS overweights deficits in patients with left versus right brain strokes.^17,18 Thus, left hemisphere strokes score 4 more points than right hemisphere strokes of similar size. The NIHSS is internally consistent, with a reasonable Cronbach’s alpha and reproducible across the intended range of users: stroke nurses, vascular neurologists, and ED physicians.^19–21 The scale is reliable when used by non-neurologists who undergo training.^20,21 The total NIHSS score can predict outcome or the presence of large vessel occlusions.^22,23 A reasonable estimate of the NIHSS can be made from chart review.²⁴

In 1995, after the publication of the Trial, the NIHSS became the de facto standard for rating clinical deficits in stroke trials. Several contemporary scales were similar^25–28 because there are few ways to put numbers to the neurological examination for the purpose of clinical research or bedside stroke severity measurement. To encourage greater use, such a scale must be short, but to capture all deficits, it must be long; to improve reliability, the scale must be simple, but to measure stroke deficit accurately, the scale must be complex; to capture important neurological findings, the scale must attempt to measure complicated concepts familiar to neurologists (eg, neglect), but during a large-scale clinical trial, non-neurologists must be able to use the scale also. The NIHSS was designed with these principals in mind.

Clinical Trials Versus Clinical Use

When using the NIHSS, it is critical to acknowledge that the scale was not designed to serve as a bedside rating tool for widespread use outside of research trials.⁶ Rather, the scale was designed to be used by investigators (MD, RN) in the setting of a clinical trial.¹⁵ The NIHSS design assumes that the user will cooperate with extensive training prior to attempting certification. The scale is intended to be used with training to assure reproducibility: when the scale is used across clinical trial sites by users of differing skill levels, the results must be reproducible.²⁹ The accuracy of the scale—whether it captures each individual patient’s deficit accurately—is secondary. Thus, the scale does not accurately reflect a patient’s coordination; gait impairment; cortical sensory function; distal motor function; memory; or cognition. This lack of accuracy was designed intentionally as a sacrifice to gain reproducibility. If one wanted to accurately capture the deficits in each individual patient, one would do a standard neurological examination and write a detailed narrative.³⁰ The accuracy of such a narrative depends heavily on the training, skill, and interest of the examiner, so the results cannot be quantified and cannot be reproduced by untrained examiners of varying skill levels. In contrast, a simplified examination scale used in serial examinations of groups of patients shows excellent characterization of the group behavior over time.³¹

To gain reproducibility and to allow non-neurologists (emergency physicians and nurses) to participate in the Trial, scoring rules were designed to facilitate reproducibility (Table 2). For example, the cardinal rule in using the NIHSS is “score what you see, not what you think.” In other words, a skilled neurologist would not down-score a patient with aphasia for failing to answer 2 questions about orientation—the neurologist would know that the aphasia prevented valid testing of orientation (item 1b; Table 1). Clinical trial designers could not assure that the non-neurologist MD, or the non-neuro-specialist RN, would do similarly in all circumstances. Therefore, the scoring rules were written to force the user to score a 1- or 2-point deficit, even in the face of obvious aphasia. This scoring rule makes little sense neurologically—the aphasia is the problem and the patient does not have stupor or delirium—but the answers to item 1b will be reproducible.^32,33 To enhance reproducibility further, several more scoring rules were written that typically strike the skilled neurologist as counterintuitive (Table 2; online-only Data Supplement).

Certification and Training

Because the NIHSS instructions include counterintuitive scoring rules, training and certification would be critical, including actual demonstration of the scoring rules on live patients. At the time (late 1980s), video technology was emerging, and training videos were being produced for ongoing clinical trials.³⁴ Prior to beginning the Trial, a training videotape and 2 certification videotapes were produced (detailed methods provided in the online-only Data Supplement). All participants in the Trial were required to view the training tape and one certification tape and score each certification patient; only after passing central review and approval were investigators certified to enroll patients. To overcome bias introduced by the video technique, we designed a scoring system that accounted for the artificial limitations of the video viewing process.¹⁵ Users who failed certification were asked to rewatch the training video and try again. After 6 months, all users were asked to view and score another certification tape to assure continued proficiency.

One intended consequence of the tapes was that new investigators could be added easily to the trial, an innovation at the time. However, an unintended consequence of this scoring system is that there were >1 correct responses to many of the case scenarios, creating an impression of leniency in the scoring.³⁵ Nevertheless, the scoring system does allow easy certification of online viewers.

Use in Trials

After the publication of the Trial in 1995, and regulatory approval of r-tPA for acute stroke in 1996, clinical trialists expressed interest in using the NIHSS for their clinical trials. Hundreds of the videotapes were produced and shipped; centralized scoring was done at Henry Ford Hospital. After a few years, the videotapes were replaced with training/certification digital videodiscs that demonstrated each NIHSS scale item and its scoring rules in detail (see detailed methods in the online-only Data Supplement). The NINDS took over responsibility for distributing the digital videodiscs to interested groups who were organizing large clinical trials, and the author provided central review and grading services using the scoring algorithm developed during the Trial.³⁶

Today, most NIHSS training and certification is performed by 3 online services (Table 3). For scoring, all services use the published NINDS algorithm, as verified by the author.³⁶ None of the vendors require the student user to view the training video, despite evidence that such training is necessary.^20,37,38 Nevertheless, online certification has been validated and is ongoing.² As of February 2016, one of the sites had certified over 500 000 different student users, most of them multiple times. Recertification is generally required annually, although some clinical trial sponsors allow longer intervals; there is no data that supports any particular recertification schedule. The Training and Certification videos have been translated into multiple languages.^39–45 Generally, the testing materials (word list, sentences, and naming card) were translated literally, but in some cases, a more rigorous process was used.⁴⁶ For use in China, entirely new video was recorded using Chinese patients and investigators.⁴³ Although ideal, reshooting the video in each country would be prohibitively expensive, so in some countries, the English video was dubbed using actors.⁴⁴

Modified Versions

There have been a few attempts to improve the accuracy of the NIHSS by removing items that lack sufficient reproducibility. The most validated modified NIHSS (Table 1) collapses items 3 and 4 into only normal/abnormal responses and eliminates the Ataxia item altogether.^16,47 The modified NIHSS is particularly well suited to applications in telemedicine.⁴⁸ In other modifications, an attempt has been made to shorten or simplify the scale or focus on a few, easy-to-teach items; although none of these shorter versions have been subjected to the same rigorous validation as the original scale, they may be useful in situations that do not require the rigor of a clinical trial.

The NIHSS contains 4 factors in formal factor analysis.^16,32 These factors represent, as intended, the 2 cerebral hemispheres. Of particular significance, each hemisphere factor resolves into cortical and subcortical factors (Figure). This result, which has been replicated, suggests that the NIHSS serves its intended purpose: numbers are generated that quantify the function of the key brain areas above the tentorium. An alternative method to report NIHSS scores would be to generate factor scores. Such an approach would overcome the known predilection of the trial toward higher scores in left-hemisphere stroke.^17,49 Factor scores do not carry clinically intuitive meaning, however, and would not be accepted easily. A frequently cited weakness of the NIHSS is the failure to capture or quantify brain stem function, although this aspect of the scale was designed intentionally: most clinical trials exclude brain stem strokes because of their infrequency and possible differing natural history.

• Download figure
• Download PowerPoint

Use of the NIHSS Outside of Clinical Trials

In addition to certifying examiners involved in clinical trials, the NIHSS has been used in demographic and epidemiological studies. A reasonably accurate NIHSS can be reconstructed from well-documented neurological examinations recorded in medical records.²⁴ Extracted NIHSS scores may not be comparable to scores recorded by certified users working in the context of clinical trials, however.

In recent years, regulatory and payer agencies have required severity descriptors in stroke patients. Baseline severity score correlates with discharge disposition, mortality, and other outcomes.^50,51 To comply, large numbers of bedside clinicians are accessing NIHSS certification online (Table 3). These users do not view the training video, so it is unclear whether or how they come to understand the scoring rules; these users may be uninterested in the scoring rules and are unlikely to be involved in clinical research.³⁸ Thus, using the NIHSS for severity scoring by such individuals seems potentially fraught with error. At a minimum, regulators should be aware that NIHSS scores generated by casual, bedside users should not be compared with scores generated in clinical trials. On the other hand, bedside users could acquire sufficient proficiency using the NIHSS to communicate with each other. A patient’s total NIHSS score portrays a vaguely accurate description of the patient equivalent to the descriptors mild, moderate, or severe stroke. In this context, it may not matter that the user does not understand the proper method to perform the scale or to use the scoring rules. A few critical steps in using the scale at the bedside are summarized in Table 2.

Because of the scoring rules, certain scores are impossible to obtain, especially at the higher end. For example, the score contains 42 possible points were a patient to score the worst on all items, but this cannot happen. In a coma patient, certain scores default to 0, for example, item 7, ataxia (Table 4), and the maximum score in a comatose patient is 39.

After the publication of major neurothrombectomy trials, there is renewed interest in using the NIHSS to select patients most likely eligible for thrombectomy.⁵² Although there is a good correlation between NIHSS and likelihood of finding an eligible large vessel occlusion, no specific cut point of the NIHSS seems optimal for field use.^22,53 Many agencies seek to use a field assessment for triaging patients to a comprehensive stroke center; at this time, neither the full scale nor any derived scale has sufficient sensitivity and specificity to be used in this way.⁵⁴ Nevertheless, a baseline NIHSS is useful in identifying patients more likely to have an eligible lesion, even though it may not be good enough for field triage in which some patients may be diverted away from appropriate resources.

Future Studies

No data exist to determine whether widespread use of the NIHSS at the bedside yields scores that are reproducible or whether users certifying without training use the scale correctly. If typical bedside use of the NIHSS today is unreliable, considerable effort will be needed to design an effective training strategy. It may be necessary to alter the online web-based training sites so that training is required before users can certify. Also, research is needed to determine how often users should be required to recertify. At the moment, annual or biannual recertification seems most common, but regulators should be aware that there is no data to support such timelines: recertification may be best if it occurs more or less often or on a progressive timeframe based on past performance. Over a longer period of time, say after 3 or 4 recertifications, perhaps it should be mandatory to rereview the training materials. Further studies are sorely needed to determine whether certified users make more errors over time, as the interval from training lengthens.

Severity adjustment of outcomes is essential in modern health care. Publicly reported outcomes (mortality, 30-day readmission) must be understood in context of stroke severity.^5,50,51 Repeatedly shown, the primary drivers of long-term outcome after stroke are initial severity—almost always quantified with the NIHSS—age, and a few comorbidities, such as diabetes mellitus. Given the profound impact of baseline stroke severity on outcome, it would seem essential that casual bedside users of the NIHSS understand the design limitations, proper technique, and scoring rules. Although the NIHSS was designed for use in clinical trials, severity scoring has grown far beyond the rigorous boundaries required of stroke research teams. More serious consideration must be given to selecting the best professionals for recording the baseline severity score at hospital admission because such scores will powerfully influence that hospital’s outcomes—many of which are publically reported.

Conclusions

The NIHSS in current use evolved from an earlier version that is no longer used. The scale now used (Table 1) was designed to be reproducible when used by physicians and nurses seeking to participate in clinical trials, and may be useful in clinical practice with appropriate training and certification. Scores for left hemisphere stroke exceed right hemisphere by four points, so severity scoring must include the side of the infarct. Online video training and certification systems are available and widely used. Use of the NIHSS by casual (nonresearch) bedside users has not been extensively validated, however, and the NIHSS should be used with caution outside of a research trial for rating stroke patients’ severity. Regulators seeking to add a severity adjustment to administrative data should approach the NIHSS with a full understanding of its limitations.

Acknowledgments

I am grateful to Karen Rapp, RN, for a critical review of the article.

Footnotes