Sensitivity, Specificity, and Sex Differences in Symptoms Reported on the 13‐Item Acute Coronary Syndrome Checklist

Introduction

Patients presenting to the emergency department (ED) with undifferentiated chest pain or other symptoms suggestive of acute coronary syndromes (ACS) account for ≈10% of all ED visits¹ and present a diagnostic challenge.^{2, 3} ACS, an umbrella term used to denote the spectrum of myocardial ischemia, includes unstable angina, non‐ST‐elevation myocardial infarction, and ST‐elevation myocardial infarction.⁴ Rapid triage for this potentially life‐threatening condition is paramount for effective time‐dependent reperfusion therapies that reduce mortality and morbidity.⁵ Though a missed ACS diagnosis is rare (2.1% to 5.3%), consequences are serious with a 2‐fold higher risk of 30‐day mortality for patients with ACS inappropriately discharged from the ED.^{3, 6} Self‐reported symptoms are the primary method by which patients communicate to clinicians the nature of their problem. Therefore, symptoms have the potential to augment traditional clinical features and risk stratification tools in predicting the likelihood of ACS.

Symptoms of ACS have been widely described in the literature.⁷ The classic symptoms of ACS, as endorsed by the American Heart Association (AHA) and the American College of Cardiology (ACC), are chest discomfort, discomfort in other areas of the upper body, shortness of breath, cold sweat, nausea, and lightheadedness.^{4, 8} These symptoms have traditionally been used to aid in risk stratification and acceleration of care for patients presenting to the ED with symptoms suggestive of ACS.⁹ Findings from studies using predictive models to assess the value of symptoms for a diagnosis of acute myocardial infarction (AMI) or ACS have varied. Some studies found that chest pain, shoulder pain, arm pain, sweating, nausea, and vomiting are predictive of AMI/ACS, but findings are not uniform across studies.^{10, 11, 12, 13, 14} Attention to classic symptoms, in particular chest pain, may disadvantage women, the elderly, and individuals with diabetes, who may experience less‐typical symptoms^{15, 16} or suffer from silent ischemia.¹⁷ However, identification of symptoms that are sensitive and specific to ACS has the potential to reduce treatment‐seeking delay and potentially expedite triage and diagnostic testing.

Most studies examining the sensitivity of symptoms associated with ACS solely recruited patients with a confirmed diagnosis,^{18, 19, 20, 21} limiting the ability to assess specificity. Specificity of ACS symptoms has been assumed to be low in women, but has not been confirmed.²² In addition, the predictive value of symptoms is more useful than sensitivity from a clinical perspective because it is the probability that a person with the symptom actually has the associated illness.²³ The objective of this analysis was to determine the sensitivity, specificity, and predictive values of 13 previously validated symptoms for a diagnosis of ACS in women and men presenting to the ED.

Methods

This analysis is part of a larger prospective, multicenter study examining the influence of sex on symptom characteristics during ACS. Patients were enrolled at 4 large medical centers; 1 in the Midwest, 2 in the Pacific Northwest, and 1 in the West regions of the United States. Three were academic medical centers, and the second Northwest site was a large community medical center. Approval from all 4 institutional review boards was received before the start of the study, and all participants gave written informed consent. Each institutional review board approved a waiver of initial consent for electronic screening of patients at triage and to collect initial symptom data before enrollment. A waiver of initial consent was granted because the main study aim was to evaluate symptoms on presentation to the ED and because the emergent nature of patients presenting with possible ACS precluded the provision of immediate informed consent. All patients were enrolled when they were deemed to be stable by the primary nurse or physician and had been transferred to a private examination room in either the ED or hospital. Initially collected symptom data were destroyed if the patient subsequently declined to participate.

Study Population

Individuals presenting to the ED between January 2011 and March 2013 with symptoms triggering an ACS evaluation and who were ≥21 years, fluent in English, and arrived by emergency medical services, private, or public transportation were eligible. Patients were excluded if they had an underlying exacerbation of heart failure (defined as BNP ≥500 pg/mL), were transferred from a hemodialysis facility, were being evaluated for a dysrhythmia, were non‐English speaking, or had cognitive impairment (defined as the inability to understand and provide written informed consent for the study).

Because most patients presenting to the ED for symptoms suggestive of ACS will be ruled out, a targeted sampling plan was implemented for this study. Patients most likely to be ruled in were identified before enrollment based on standard ECG and troponin criteria.²⁴ Those patients with any ECG changes suggestive of ischemia and/or with a troponin level outside the referenced norm for the institution were approached for enrollment. Ischemia was defined as new ST elevation at the J point ≥0.1 mV in 2 contiguous leads and/or new horizontal or down‐sloping ST depression ≥0.05 mV in 2 contiguous leads and/or T inversion ≥0.1 mV in 2 contiguous leads with prominent R wave.²⁴ Discharge diagnoses (ACS versus non‐ACS) were based on the clinical judgment of the ED physician for patients discharged from the ED and the attending physician for patients who were admitted. Physicians remained unaware of the symptom data collected by the researchers. Discharge diagnoses were abstracted from the medical record by trained research associates who were clinical experts and were also blinded to the research data. Equivocal or discrepant discharge diagnoses were adjudicated by 3 of the authors (M.D., H.D., and A.R.). Clinical data on self‐reported comorbid factors, such as diabetes and hypertension, were confirmed from the medical record. If there were discrepancies, only the data from the medical record were used for the analyses.

Measures

The ACS Symptom Checklist is a 13‐item validated instrument that measures symptoms of ACS.^{19, 25} Participants indicate whether the symptom is present or absent. Symptoms not appearing on the checklist can be recorded in a blank space marked “other.” Each symptom is analyzed individually, and there is no summary score. The ACS Symptom Checklist is derived from the Symptoms of Acute Coronary Syndromes Index (SACSI). The SACSI, a reliable and valid instrument, has been tested in previous studies.^{19, 25} Participants indicated whether the symptom was present or absent. Symptoms not listed on the checklist were recorded in a blank space marked “other.” A content validity index of 1.00 (P<0.05) using Lynn's formula²⁶ was calculated based on responses from 11 experts. One item, heat sensation, was judged irrelevant and removed from the original 14‐item checklist.

Patient baseline characteristics were collected using the ACS Patient Information Questionnaire. This demographic and clinical questionnaire was designed using the standardized reporting guidelines recommended for studies evaluating risk stratification of ED patients with potential ACS.²⁷ These guidelines were established by the Multidisciplinary Standardized Reporting Criteria Task Force and are supported by the Society for Academic Emergency Medicine, the American College of Emergency Physicians, the AHA, and the ACC. The purpose of the questionnaire is to establish standardized reporting criteria that will allow for easier comparisons across studies and also facilitate meta‐analyses. Functional status was measured with the Duke Activity Status Index (DASI).²⁸ The DASI is a brief 12‐item instrument that measures functional capacity. Scores range from 0 to 58.2, with higher scores representing better physical functioning. The items on the scale are weighted to reflect metabolic energy expenditure and correlate highly with peak VO₂ (r=0.80; P<0.0001)²⁸ in patients with ACS,²⁹ ischemic heart disease,³⁰ heart failure,³¹ and revascularization procedures.³²

Procedures

A trained member of the study research team completed the ACS symptom checklist shortly after the patient was triaged in the ED. Patients were enrolled between 0700 and 2300 every day. In most cases, symptoms were assessed within 15 minutes of ED presentation. Patients were then formally approached by the research staff for enrollment after they were deemed stable and placed in an ED examination room or in their hospital room. The study purpose was then explained, and once the patient provided written informed consent, additional clinical and individual characteristics were recorded.

Statistical Analysis

To address power concerns for differences in symptoms between patients ruled in and ruled out for ACS, a sample size of 261 per group (n=522) was needed for 80% power to detect a small effect size (d=0.25) using ANOVA with a 0.05 2‐sided level of significance. Because the primary purpose of the study was sex differences in symptoms, we aimed to enroll an equal amount of women and men. Study data were entered into SAS. Significance was set at P<0.05 for all statistical procedures. Demographic and clinical characteristics were described for the total sample and also compared by groups defined by ACS diagnosis and sex using chi‐square tests for independence, independent samples t tests, and Wilcoxon rank sums tests.

Sensitivity was defined as the probability of the presence of a symptom among patients with a confirmed diagnosis of ACS. Specificity was defined as the probability of the absence of a symptom among patients ruled out for ACS. Sensitivity and specificity were computed for each of the 13 symptoms. Sensitivity and specificity are usually calculated as simple proportions using a 2×2 cross‐tabulation of symptom (present, absent) versus diagnosis (ACS, no ACS); however, data collection sites varied on the proportion of patients ruled in for ACS (P<0.0001). Therefore, mixed‐effects logistic regression models, including random intercepts to control for site differences, were used to estimate the sensitivity and specificity of each symptom for a confirmed ACS diagnosis. Models were tested for the entire sample and then stratified by sex. A sensitivity and specificity of ≥60% were considered to be moderately high for this sample based on previous studies of the sensitivity and specificity of symptoms for a diagnosis in a variety of conditions.^{33, 34, 35, 36} This range is also consistent with sensitivities and specificities reported in Bruyninckx's meta‐analysis.³⁷ Similar logistic regression models were used to estimate the odds ratios (ORs) of each symptom as a predictor of an ACS diagnosis controlling for age, obesity, diabetes, functional status, and sex. A sex by symptom interaction term was evaluated to determine whether the relationship between symptoms and diagnosis varied between women and men. Potential covariates were chosen based on sex differences in previous studies and because age, obesity, functional status, and diabetes can confound the symptom experience.^{38, 39}

Results

Sample Characteristics

The sample (n=736) included 301 patients (40.9%) ruled in and 435 (59.1%) ruled out for ACS. A total of 10 896 patients were screened. Of the 1005 patients eligible to participate, 269 (26.8%) declined ( Figure). The most common reason given was stress, discomfort, or fatigue. Of the 301 patients who ruled in for ACS, 9.6% had normal troponins and a normal ECG, 24.9% had normal troponins and an abnormal ECG, 12.6% had elevated troponins and a normal ECG, and 52.9% had elevated troponins and an abnormal ECG. Patients were predominantly Caucasian (70.6%) and had a mean age of 59.7±14.2 years. Overall, participants ruled in for ACS were older than those ruled out for ACS (61.3±12.2 versus 58.6±15.4 years, P=0.013; range 21 to 98), were more likely to have diabetes (31.8% versus 24.6%; P=0.034), more often male (74.4% versus 55.2%; P=<0.0001), and more likely to be in the middle‐income ranges. ACS was more common among the income group, ranging from $20 000 to $49 999, whereas a plurality of patients without ACS were in the lowest income group (P=0.001). Men had higher rates of current smoking than women (23.3% versus 14.7%; P=0.003). Demographic data appear in Table 1 and clinical characteristics are summarized in Table 2.

• Download PowerPoint

Symptom Occurrence by Diagnosis and Sex

Patients ruled in for ACS were more likely to experience chest pain (71% versus 61%; P=0.006), but less likely to experience palpitations (19% versus 28%; P=0.008), upper back pain (19% versus 29%; P=0.001), shortness of breath (46% versus 60%; P<0.0001), unusual fatigue (34% versus 49%; P<0.0001), and lightheadedness (36% versus 46%; P=0.006), compared to patients ruled out for ACS (Table 3). Women ruled in for ACS were more likely to report arm pain (47% versus 32%; P=0.021), compared to women ruled out for ACS. Men ruled in for ACS were more likely to report chest pressure (63% versus 54%; P=0.035) and chest pain (72% versus 60%; P=0.005) and less likely to report upper back pain (13% versus 24%; P=0.004), shortness of breath (41% versus 59%; P<0.0001), and unusual fatigue (32% versus 48%; P=<0.001). The mean number of symptoms ranged from 5 to 6 across groups, and patients in each group reported the full range of symptoms (1 to 13). There were no differences between patients ruled in for ACS and those ruled out for ACS or between women and men for mean number of symptoms.

Sensitivity and Specificity of Symptoms for a Diagnosis

Sensitivities for individual symptoms on the checklist ranged from 27% to 67% for women and 14% to 72% for men (Table 4). Chest pressure, discomfort, and chest pain demonstrated the highest sensitivity for ACS in both women (66%, 66%, and 67%, respectively) and men (63%, 69%, and 72%, respectively). Specificities ranged from 33% to 78% for women and 34% to 78% for men. Six symptoms in particular—shoulder pain, sweating, palpitations, upper back pain, arm pain, and indigestion—had higher specificities (>60%), indicating that those patients who did not experience those symptoms were likely to be ruled out for ACS. Nausea, however, was specific for men only.

Predictive Value of Symptoms for ACS Diagnosis

Each symptom was tested as a predictor of ACS diagnosis in models adjusted for data collection site, age, obesity, diabetes, and functional status. A symptom by sex interaction term was also tested because there have been previous reports of sex differences in symptoms.^{20, 21} There were significant interactions for sex and shoulder pain (P=0.034) and shortness of breath (P=0.004) (Table 5), meaning that the relationship between the symptom and ACS diagnosis differed by sex. Analyses of symptoms stratified by sex indicated that shoulder pain (OR=2.53; 95% CI=1.29 to 4.96) and arm pain (OR=2.15; 95% CI=1.10 to 4.20) were predictive of an ACS diagnosis for women, but not men (OR=1.11; 95% CI=0.67 to 1.85; OR=1.21; 95% CI=0.74 to 1.99, respectively). Shortness of breath was predictive of a non‐ACS diagnosis for men (OR=0.49; 95% CI=0.30 to 0.79), but was not predictive of a diagnosis for women (OR=1.36; 95% CI=0.68 to 2.70).

Discussion

The findings that all 3 chest symptoms (pressure, discomfort, and pain) were sensitive, but not specific, for a diagnosis of ACS, whereas shoulder and arm pain were predictive of an ACS diagnosis for women only, are important findings. Goodacre et al¹² also found that pain radiating to either arm was predictive of ACS. However, Goodacre's study did not include sex‐stratified analyses in the results. Swap and Nagurney¹⁰ also found that chest pain radiating to one or both shoulders or arms increased the likelihood of an ACS diagnosis. Lack of sex‐stratified analysis has been a continuing problem in understanding the influence of sex on ACS symptoms.⁷ Earlier findings have demonstrated that older patients and patients with diabetes report less chest pain during ACS.¹⁵ Because women are, on average, 10 years older than men⁴ when they develop ACS, shoulder and arm pain (particularly in women) may serve as additional prompts that aid emergency medical services and triage personnel in identifying patients with ACS—particularly in the absence of chest symptoms. Whether these findings apply to women under age 55 who are at higher risk for mortality and morbidity, but at lower risk for development of ACS, requires further study.⁴⁰ As expected, chest pressure, discomfort, and pain were sensitive for a diagnosis of ACS in both women and men.

Patients ruled in for ACS were older than those ruled out for ACS, more likely to have diabetes, and more often male. This is consistent with Herlitz et al⁴¹ (OR=1.97; 95% CI=1.30 to 2.99) and Edwards et al⁴² (relative ratio=1.48; 95% CI=1.00 to 2.18), who found male sex to be an independent predictor of an ACS diagnosis. In a large meta‐analysis, Haasenritter et al⁴³ found that age (likelihood ratio [LR]=1.44; 95% CI=1.19 to 1.73), diabetes (LR=1.68; 95% CI=1.35 to 2.09), and male sex (LR=1.17; 95% CI=1.08 to 1.27) were associated with an ACS diagnosis.

Income also differed significantly between the ACS and non‐ACS groups, but in an unanticipated way; patients in the middle‐income levels were more likely to have ACS than those in the lower‐income levels. It is unknown why individuals in the middle‐income ranges were more likely to be ruled in for ACS. Perhaps, the middle‐income groups lack health insurance or are underinsured. This requires further study, particularly in view of the Affordable Health Care Act.⁴⁴

Shoulder pain, sweating, palpitations, upper back pain, arm pain, and indigestion had moderately high specificities for a non‐ACS diagnosis (≥60%) in both women and men. Nausea had a high specificity for men only. These findings support the editorial by Canto et al⁴⁵ suggesting that it is time to standardize the collection of ACS symptoms in patients presenting to the ED in order to determine the significance of sex differences in symptoms of ACS. Lack of specificity of chest symptoms for a diagnosis of ACS requires further study, and use of a standardized tool would facilitate comparisons across studies. In addition, lack of specificity for chest symptoms could contribute to delays in patient decisions and in diagnosis because noncardiac chest pain is a common and costly occurrence.⁴⁶ Specificities in the 60% to 70% range are not high enough to preclude the evaluation of ECGs and troponins in patients presenting to the ED with symptoms suggestive of ACS, but they may be high enough to design evidence‐based public health messages for the public and especially for patients at risk for ACS.

Absence of shortness of breath was predictive of a non‐ACS diagnosis in men. This further creates a diagnostic conundrum in the ED because most patients are undifferentiated on presentation, and streamlined care is an expectation for potentially life‐threatening conditions such as ACS.⁴⁷ Although there were no sex differences in reports of chest pain, men with ACS were more likely to experience chest pain, compared to men without ACS. Reports of sex differences in chest pain have varied in large database studies and smaller cohort studies. Some investigators have found that women are less likely to report chest pain^{7, 21} and some have reported no differences between women and men.^{19, 48} Our results suggest that low specificities of symptoms are also likely to impact ED symptom‐based triage. The current approach, in which a wide net is cast for ACS, appears necessary to avoid missing a true ACS diagnosis.

The mean number of symptoms experienced by participants was high, which is important when planning public health messaging. Being knowledgeable about, and alert for, a single symptom such as chest pain is not sufficient to make an informed decision about proper care seeking when experiencing possible ACS. The public should be informed that multiple symptoms are likely during ACS. Because the mean number of reported symptoms exceeded 5 for both women and men, it is likely that the assessment of individual symptoms is not sufficient to improve clinical assessment and decision making.

Strengths

This prospective study was well powered, and symptoms reported directly by the patients were recorded shortly after presentation to the ED. The availability of real‐time self‐report of dynamic symptoms effectively eliminates recall bias and increases the internal validity of the findings,⁴⁹ which has been a limitation of previous ACS studies. Our 13‐item validated checklist contains symptoms that are empirically derived from large heterogeneous samples of patients, takes <1 minute to complete, and is suitable for research and clinical practice. The demographic and clinical characteristics of our patients suggest that this sample was representative of the US population and that there were minimal baseline differences between patients with and without ACS. Finally, we adjusted for factors well known to affect symptoms as reported in the literature.

Limitations

Sampling bias is a limitation to the study. We enrolled only patients whom ED nurses and physicians deemed to be at risk for ACS. Therefore, true ACS patients may have been missed if an ACS diagnosis was not considered. Also, in order to enroll a sufficient number of patients with confirmed ACS, the sample was enriched by targeting patients with an abnormal ECG or troponin and thus may not be representative of the individuals presenting to triage. Further, though an abnormal ECG and troponin were part of the sampling plan, in the final adjudication (discharge diagnosis) clinicians felt that nearly 10% of patients with normal ECG and troponin levels did have ACS. Our 13‐item validated checklist may have missed other symptoms; however, patients were asked if they were experiencing any other symptoms not contained on the list. Whereas Canto⁴⁵ has called for the use of a more comprehensive symptom instrument, our goal was to use a validated symptom tool that was suitable for use as a clinical assessment tool in prehospital and hospital triage as well as in primary care and research. Finally, patients were only enrolled between the hours of 0700 and 2300. Hence, findings may not be generalizable to patients that presented between 2300 and 0700.

Whereas we believe self‐report to be strength of the study, some have considered it a limitation because there is no way to externally validate the symptom event. However, Justice et al³³ noted that patients, not providers, experience symptoms and so are best able to describe them. Self‐reported symptoms have been independently associated with quality of life and activities of daily living in patients with acquired immunodeficiency virus.⁵⁰ Self‐report of symptoms may be considered superior to symptoms abstracted from the medical record because there is no way of knowing how accurate the medical record is or to independently validate which symptoms were actually reported by patients. In addition, prudence is called for in generalizing these findings outside of the ED. Patients presenting to the ED are a select group that made a conscious choice to seek care for symptoms judged to be potentially serious. Therefore, sex differences in symptoms may be attributable to selection bias in behaviors associated with a decision to seek assistance, rather than true pathophysiologic or psychosocial difference in symptoms. A considerable number of patients with undifferentiated chest pain were excluded from the study by design because the main aim was to evaluate sex differences in symptoms among patients with confirmed ACS. This may have particularly influenced the findings on specificity of individual symptoms. Finally, the predictive power of ORs between 1.0 and 2.0 may be limited and does not support discharging low‐risk patients on the basis of symptoms alone without further testing, such as ECG and cardiac biomarkers.¹⁰

Conclusions

There were more similarities than differences in symptom predictors of ACS for women and men. Women were twice as likely to report arm and shoulder pain, compared to men. Sex differences in symptoms require further study to help guide patients’ treatment seeking decisions. Although sex differences were minimal, shoulder pain and arm pain may be key symptoms that improve clinical prediction of ACS in women. These symptoms may help guide the clinician in deciding the extent of diagnostic workups.

Sources of Funding

This work was funded by the National Institute of Nursing Research (R01NR012012).

Disclosures

None.

Footnotes