The Rapid Arterial oCclusion Evaluation (RACE) scale accuracy for diagnosis of acute ischemic stroke in emergency department – A multicenter study

Abstract

Objective

It seems that the available data on performance of the Rapid Arterial oCclusion Evaluation (RACE) as a prehospital stroke scale for differentiating all AIS cases, not only large vessel occlusion (LVO), from the stroke mimics is lacking. As a result, we intend to evaluate the accuracy of the RACE criteria in diagnosing of AIS in patients transferred to the emergency department (ED).

Method

The present study was a diagnostic accuracy cross-sectional study during 2021 in Iran. The study population consist of all suspected acute ischemic stroke (AIS) patients who transferred to the ED by emergency medical services (EMS). A 3-part checklist consisting of the basic and demographic information of the patients, items related to the RACE scale, and the final diagnosis of the patients based on interpretation of patients’ brain MRI was used for data collection. All data were entered in Stata 14 software. We used the ROC analysis to evaluate the diagnostic power of the test.

Result

In this study, data from 805 patients with the mean age of 66.9 ± 13.9 years were studied of whom 57.5% were males. Of all the patients suspected of stroke who transferred to the ED, 562 (69.8%) had a definite final diagnosis of AIS. The sensitivity and specificity of the RACE scale for the recommended cut-off point (score ≥ 5) were 50.18% and 92.18%, respectively. According to the Youden J index, the best cut-off point for this tool for differentiating AIS cases was a score > 2, at which sensitivity and specificity were 74.73% and 87.65%, respectively.

Conclusion

It seems that, the RACE scale is an accurate diagnostic tool to detect and screen AIS patients in ED, Of course, not at the previously suggested cut-off point (score ≥ 5), but at the score > 2.

Introduction

Stroke is one of the most common leading cause of both disability and death worldwide, and more than 80% of stroke cases are ischemic types globally [1]. Despite of introduction of intravenous thrombolytic therapy and also mechanical thrombectomy for curing acute ischemic stroke (AIS) patients, due to various reasons, less than 5% of eligible AIS patients received such treatment in the therapeutic window period [2,3,4]. It seems that due to the delay in referring patients, considerable number of AIS patients lose the chance to receive treatment [5, 6]. Therefore, various interventions have been considered and applied by health care systems during the last decades to reduce the time between AIS onset and starting an appropriate management [7,8,9]. One of the most important measures was the reduction of wasting time both at the pre-hospital phase and also after admission to the emergency department (ED) [10, 11]. Several stroke screening tools have been designed in this regard that can accurately predict stroke [11], that mostly applied in prehospital setting and their performance has been rarely evaluated in ED environment. Rapid Arterial oCclusion Evaluation (RACE) scale is one of such tools that designed by Pérez de la Ossa et al. not only for screening the stroke cases, but also to differentiate large vessel occlusion (LVO) cases than the others [12]. Despite of high sensitivity and negative predictive value in the primary derivation/validation study, various results were further reported by other investigators about it [13, 14]. It seems that the available data on its performance is still under debate and more focused on differentiating LVO cases and not all AIS patients, and to the best of our knowledge, no one has tested its performance in the ED till yet. We believed that it needs more study to resolve the current controversies, and more importantly, accuracy of RACE for differentiating all AIS cases from the stroke mimics should be evaluated. As a result, we intend to evaluate the accuracy of the RACE criteria in diagnosing of AIS in patients transferred to the ED.

Methods

Study design and population

The present study was a diagnostic accuracy cross-sectional study that was conducted from 15 to 2021 until 28 December 2021. The research environment was the ED of research, educational and medical centers in Tehran, Isfahan, and Ahvaz cities, in Iran. The study population was all patients who transferred to the ED by EMS with suspected AIS diagnosis.

According to a study by Pérez de la Ossa et al. [12], assuming a sensitivity of 100% and a specificity of 44%, the prevalence of AIS (patients suspected of stroke transferred to an emergency department) was 45%, estimation accuracy was 95% and error was 5%. The sample size was 841 people.

Data collection

For this purpose, a checklist consisting of three sections was prepared. The first part related to the basic and demographic information of the patients including age, gender, history of underlying diseases, etc.; the second part consisted of 5 items related to the RACE scale [facial palsy (scored 0–2), arm motor function (0–2), leg motor function (0–2), gaze (0–1), and aphasia or agnosia (0–2); each person can obtain between zero to 9 points]; and the third part included the final diagnosis. The diagnosis of AIS was made based on the findings of physical examination and also interpretation of patients’ brain magnetic resonance imaging (MRI) which is considered as the gold standard method of AIS diagnosis in the current study.

Sampling was conducted prospectively. From the beginning of the study period, the checklist was included in the patients’ hospital files to assess proper neurological examination of patients with any neurological complaints. A postgraduate year (PGY)-2 neurology resident used the RACE scale to evaluate the suspected patients and recorded all documents in the checklist when the patient had been arrived in the ED. A double check on the checklist was also performed by a PGY-3 emergency medicine resident.

Statistical analysis

The data collected by patient evaluation according to RACE scale was compared with the results of the physician’s diagnosis, as gold standard. We used the ROC curve and the area under the ROC curve (AUC) to evaluate the differentiated power of the RACE scale. The accuracy indices of RACE scale, like sensitivity, specificity, positive and negative predictive value calculated for different cut-off point. To determine the best cutoff point, we used the Youden index and the maximum of vertical distance of ROC curve from the point (x, y) on diagonal line. Also, we used of the univariable and multivariable logistic regression model for assess of effect size of any RACE criteria scale in AIS diagnosis. All statistical analysis conducted with Stata 14 software, College Station, TX: StataCorp LP. The statistical analysis conducted menu-base if available in the software and commands are used to perform ROC analysis, like roctab, dtroc, and cutpt (option: near and youden) syntax. Also, we used of tabulate data for calculate of specifying detail of indices of RACE scale performance for each of the possible cut-points.

Results

Baseline findings

In this study, data from 805 patients with suspected AIS transferred to the ED were collected. There were 463 (57.5%) males and 342 (42.5%) females among the studied patients. The mean age ± standard deviation (SD) of the patients was 66.9 ± 13.9 years, and the age range of the patients varied between 6 and 95 years. Table 1 presents some of frequency distribution of risk factors and underline diseases in the study patients.

Table 1 Frequency distribution of risk factors and underline diseases in the study patients

The mean blood glucose level of the patients on arrival to the ED was 148.2 ± 68.1 mg/dL, and its range varied between 27.0 and 560.0 mg/dL. The mean time interval between the onset of symptoms and the arrival at the ED was 8.4 ± 9.9 h and median was 2.5 h, that varied between half an hour to 72 h.

Of all the patients suspected of stroke who transferred to the ED, 562 (69.8%) had a definite final diagnosis of AIS, and the rest of 243 (30.2%) patients were negative for this diagnosis. In other words, the prevalence of AIS in transferred patients was 69.8% (95% CI: 66.6 to 73.0%). Patients with a positive diagnosis of AIS were older than patients with a negative diagnosis. The mean age of the patients with and without AIS was 68.1 ± 13.2 years and 64.1 ± 14.9 years, respectively (p < 0.001). The prevalence of AIS was also higher in men than women (73.9% vs. 64.3%; p = 0.004). Besides that, this prevalence of AIS diagnosis was higher in patients with a history of ischemic heart disease than patients without a similar history (74.9% vs. 67.1%; p = 0.021).

Detailed results of RACE criteria

Based on the total RACE score, 221 patients (27.5%) had a score of zero. The prevalence of AIS in patients with a zero score was 29.4%, while this value was 100% for patients with a score of 8 and 9 (Table 2).

Table 2 Frequency distribution (prevalence) of AIS in the patients with suspected stroke transferred to emergency department by RACE score

Assessment of the relationship between each item of the RACE criterion showed that all items of this criterion have a statistically significant relationship with the final diagnosis of stroke. So that the frequency of disorders in each item was higher in patients who had a definitive diagnosis of AIS than the others (Table 3).

Table 3 Frequency distribution and relationship of each RACE criteria with a definitive diagnosis of AIS in the patients with suspected stroke transferred to emergency department (n = 805)

The accuracy of AIS diagnosis, and best diagnostic cut-off points

The area under the ROC curve of the RACE scale in the diagnosis of AIS was 0.86 (95% confidence interval (CI): 0.83 to 0.88; p < 0.001). (Fig. 1).

Fig. 1

ROC curve for RACE scale in the diagnosis of AIS in patients suspected of stroke transferred to emergency department

The recommended cut-off point for the RACE tool for stroke screening in suspected patients was a score greater than 4. Based on this cut-off point, 62.8% of the patients had an appropriate classification. In other words, at this cut-off, 19 patients (2.4%) were negatively screened for AIS by mistake (false negative), and 280 patients (34.8%) were positively screened for AIS incorrectly (false positive). However, the highest percentage of correct classification (81.5%) and consequently the lowest error in screening according to RACE tool related to the cut-off point greater than one (Table 4).

Table 4 Frequency distribution of true and false positives and negatives in the diagnosis of AIS based on various cut-off points of the RACE scale and the percentage of correct classification

The sensitivity and specificity of RACE scale for the recommended cut-off point (score > 4) were 50.18% and 92.18%, respectively. According to the Youden J index and maximum of vertical distance, the best cut-off point for this tool was a score greater than 2, at which sensitivity and specificity were 74.73% and 87.65%, respectively. In the best cut-off point the positive and negative predictive values ​​of the RACE tool were 93.3% and 60.0%, respectively (Table 5).

Table 5 Accuracy and predictive value indicators for RACE tool at different cut-off points for screening of AIS

Logistic regression results of RACE scale

Logistic regression analysis showed that among the RACE criteria in univariable analysis, all criteria unaccompanied had a significant relationship with AIS diagnosis, to put it another way, the risk of AIS in patients for whom each of the criteria was positive was significantly higher and this relation for the “Head & Gaze Deviation” criterion was the highest, and the risk of AIS in the patients who were positive for this criterion was more than 28 times than that of the patients who were not.

Multivariable regression showed that the variables “Arm Motor Function” and “Head & Gaze Deviation” had the highest and most significant relationship with AIS diagnosis. So, the risk of AIS for the patients with severe “Arm Motor Function” compared to normal ones, regardless of the status of other variables, was 20.6 times higher. The results of multivariable analysis showed that among the RACE criteria, the “Leg Motor Function”, after removing the confounding effect of other variables, had no significant relationship with AIS. Therefore, this criterion has the lowest screening value among other RACE criteria (Table 6).

Table 6 Univariable and multivariable logistic regression of RACE criteria in the diagnosis of ischemic stroke in patients suspected of stroke transferred to a hospital emergency department

Discussion

In this study, we assessed the accuracy and other statistic characteristics of the RACE scale in terms of AIS diagnosis for patients with symptoms of stroke referring by EMS to the ED, which showed that the RACE scale is a helpful and accurate diagnostic tool to detect and screen AIS patients. As it was mentioned, the primary derivation/validation study on the RACE scale was performed for differentiated LVO cases, and not all AIS patients [12]. But here, we assessed its accuracy to differentiating, not only LVO cases but also all AIS cases; so the findings should be discussed cautiously, keeping this important issue in mind. Furthermore, in this study, the RACE scale which designed for using in the prehospital settings, was evaluated at the time of the patient arrival in ED. Therefore, we did not examine the validity of the RACE scale as a prehospital scale. Rather, we investigated the performance of this tool for diagnosis of AIS in ED.

The best cut-off point proposed in original article of RACE was score ≥ 5 for differentiating LVO cases [12], but we found that at this cut-off, the criteria had a sensitivity as low as 43.59, and a specificity of 95.47 for differentiating all AIS cases. While we suggest score > 2 as the best cut off for differentiating all AIS cases and improving the sensitivity of the RACE tool that showed a sensitivity of 74.73 and a specificity 87.65 at this cut-off point.

Carrera et al. (that seems to be participated in the primary derivation/validation study on the RACE scale [12]) conducted another two studies on this scale [13, 15]. First they tried to assess predictive value of modified versions of the RACE scale and defined 7 simpler versions of this scale, in this regard; but finally concluded that the original RACE has priority upon all modified versions [13]. Thereafter, they performed another study to revalidation of the RACE scale, in which they confirmed the scale’s accuracy to identify AIS patients with LVO [15]. In both studies by Carrera et al., and also all other conducted studies, the investigators considered the score ≥ 5 as the cut off of their analysis [13,14,15,16,17], and we cannot find any paper, in which somebody tried to assess to find another cut-off with aim of using this test for screening all AIS cases and not just LVO ones.

Thavarajah et al. [18] recently published an article in which they determined that the RACE scale has an acceptable discriminative power for identifying LVO patients in USA, however the reported statistical characteristics was lower than that of reported by the main study conducted in Spain [12] (sensitivity of 0.71 vs. 0.85, specificity of 0.65 vs. 0.68). However, we found even a lower sensitivity, meanwhile, a higher specificity than that of reported by United States-based study [18] and also Spanish-based ones [12, 13, 15].

In a multicenter, prospective, observational cohort study, Duvekot et al. [19] compared 8 prehospital stroke scales to detect LVO cases and reported that of all assessed scales, the AUC for RACE was highest and it was equal to 0·83 (95%CI: 0·79 − 0·86). While, we found that the AUC of the RACE scale in the diagnosis of AIS was 0.86 (95%CI: 0.83–0.88).

The results of logistic regression analysis in this study showed that among the variables of the RACE scale, “Arm Motor Function” and “Head & Gaze Deviation” have a larger effect size than other variables for predicting AIS. In the clinic, the importance of these two variables, or in fact, the findings from the physical examination of patients referred to ED in terms of stroke diagnosis, is very high. These findings are in line with other studies, in which, it has also been claimed that patients who are positive for these variables have a higher chance of having a definite diagnosis of AIS than those patients who are negative for these variables [13, 20].

All in all, we found that, if the RACE would be considered by a health system, as a stablished AIS diagnostic clinical tool, the cut-off > 2 would be more suitable. If the referral stroke centers for AIS cases could be differ based on equipment and availability of thrombectomy facilities, then proposed cut-off ≥ 5 should be considered. But such a facilities are not frequently available in most low and middle-income countries, so it is better to choose a proper clinical tool for stablishing in EMS system, based on each country situation, that can accurately differentiate AIS from none-AIS cases at the first step.

Limitations

We did not define and categorize the diagnosis of not AIS cases who studied in this survey, and we could not report the stroke mimics that would be valuable. Also, we did not divide the LVO cases from all AIS cases, so we can not exactly validate the defined RACE scale in its original article.

Future directions

To the best of our knowledge, almost all of previous studies were conducted to evaluate the validity or accuracy of the RACE scale, comparing its statistical indexes with other similar scales, or its performance in real practice for detecting LVO cases. We cannot find any paper, in which somebody tried to assess to find another cut-off with aim of using this test for screening all AIS cases and not just LVO ones. Therefore, we suggest other investigators to consider this point in their future researches.

Conclusion

It seems that, the RACE scale is an accurate diagnostic tool to detect and screen AIS patients in ED, Of course, not at the previously suggested cut-off point (score ≥ 5), but at the score > 2.