A machine learning model to predict critical care outcomes in patient with chest pain visiting the emergency department

Abstract

Background

Currently, the risk stratification of critically ill patient with chest pain is a challenge. We aimed to use machine learning approach to predict the critical care outcomes in patients with chest pain, and simultaneously compare its performance with HEART, GRACE, and TIMI scores.

Methods

This was a retrospective, case-control study in patients with acute non-traumatic chest pain who presented to the emergency department (ED) between January 2017 and December 2019. The outcomes included cardiac arrest, transfer to ICU, and death during treatment in ED. In the randomly sampled training set (70%), a LASSO regression model was developed, and presented with nomogram. The performance was measured in both training set (70% participants) and testing set (30% participants), and findings were compared with the three widely used scores.

Results

We proposed a LASSO regression model incorporating mode of arrival, reperfusion therapy, Killip class, systolic BP, serum creatinine, creatine kinase-MB, and brain natriuretic peptide as independent predictors of critical care outcomes in patients with chest pain. Our model significantly outperformed the HEART, GRACE, TIMI score with AUC of 0.953 (95%CI: 0.922–0.984), 0.754 (95%CI: 0.675–0.832), 0.747 (95%CI: 0.664–0.829), 0.735 (95%CI: 0.655–0.815), respectively. Consistently, our model demonstrated better outcomes regarding the metrics of accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F1 score. Similarly, the decision curve analysis elucidated a greater net benefit of our model over the full ranges of clinical thresholds.

Conclusion

We present an accurate model for predicting the critical care outcomes in patients with chest pain, and provide substantial support to its application as a decision-making tool in ED.

Background

Globally, chest pain of acute-onset is one of the most common presenting complaints in the emergency department (ED). It represents approximately 6 to 9 million visits per year in the USA [1, 2]. In this group of patients, initial assessment is guided by vital signs, ECG findings, levels of cardiac enzymes, and estimation of established risk scores. However, this initial risk stratification remains insufficient [3], and contributes to crowding of ED and delay in patient care, ultimately resulting in greater morbidity and mortality [4]. Thus, in an overcrowded ED with limited resources, it is essential to identify critically ill patients presenting with chest pain and take appropriate measures for the preferential management of these patients [5].

In the last two decades, an evolving literature related to the identification of a wide range of critically ill patients has emerged. Previous studies have developed models for predicting the clinical deterioration of patients admitted in wards. They used mortality, cardiac arrest, and transfer to intensive care units (ICUs) as their clinical outcomes, however, they could achieve only moderate performance [6, 7]. The aim for preparing the Acute Coronary Treatment and Intervention Outcomes Network (ACTION) ICU score was to predict the complications requiring ICU care in patients with non-ST elevation myocardial infarction (NSTEMI), but reported low accuracy both in development and externally validated cohorts, thereby resulting in restricted use in clinical prediction models [8, 9]. Several established clinical outcome scores have been used for risk stratification of patients with chest pain presenting to the ED, including the History, Electrocardiography (ECG), Age, Risk factors, and Troponin (HEART) [10]; the Thrombolysis in Myocardial Infarction (TIMI) [11]; and the Global Registry of Acute Coronary Events (GRACE) score [12]. The commonly used prediction outcome is major adverse cardiovascular events (MACE) [13], namely myocardial infarction, percutaneous coronary intervention (PCI), coronary artery bypass graft (CABG), coronary artery stenosis, cardiac arrest, all-cause mortality, etc. However, there are certain differences and partial overlap in the critical care outcomes of these scores. TIMI and GRACE are time consuming and only applicable in patients with acute coronary syndrome (ACS) [14, 15], thus leading to suboptimal use in patients with chest pain of various origin. Amongst various scores, the HEART score is most accurate and widely used for risk stratification of patients with chest pain [16]. It is used for safe discharge of low-risk patients [17], or identifying high-risk patient for occurrence of MACE [18]. Considering all these facts, it can be concluded that a little or no attention is being paid to the prediction of outcomes in critically ill patients presenting with chest pain. Thus, a great challenge lies ahead in constructing a promising prediction model to identify this group of patients.

In order to improve the risk predictive ability, amongst the patients presenting to ED, various machine learning (ML) algorithms (such as support vector machine [19], neural network [20,21,22], random forest [21, 22], gradient-boosted decision tree [21, 22], and least absolute shrinkage and selection operator (LASSO) regression [21, 22]) have been used and demonstrated to have a satisfactory performance. Despite the advancement in the ML algorithms, major drawback of majority of these approaches is the lack of physiological sense due to the absence of a visual model [23], which may in turn result in dissatisfaction amongst and reduced use by the healthcare workers. Nevertheless, it is reported that logistic regression with LASSO regularization (LASSO regression) may address this gap. Based on the ML approach, LASSO regularization shrinks the regression coefficients toward zero, effectively selects the important predictors, and improves the interpretability of the model. With the use of this advance technology, we attempted to produce a formula, by using logistic regression, and provided a mechanistic model by nomogram.

The primary objective of this study was to develop a ML model “LASSO regression model”, using routinely available clinical features in patients with chest pain, and to accurately predict the outcomes in critically ill patients presenting at ED. Moreover, we compared the prediction performance of the LASSO regression model with the reference score.

Methods

Study design and setting

This was a retrospective, case-control study performed in the Fujian Provincial Emergency Center, an oldest and largest tertiary care hospital in Fujian, China. In our set-up, about 500–700 patients (including 10–12 patients with chest pain) visit the ED clinics daily, and about 50–60 patients (including 4–8 patients with chest pain) are admitted daily in the first aid room. In the present study, we included the patients with chest pain who received treatment in the first aid room. By using the National Emergency Triage Guidelines of China, with 4 levels, all patients visiting the ED were initially triaged by nurses [24]. As per the Triage Guidelines, Level 1 included the most critically ill patients that required attention in first aid room without delay. Thus, they required maximum allocation of resources, healthcare staff, and equipment for the initial management. Level 2 included critically ill patients without any danger of imminent collapse, but required to be contained in the first aid room for further examination and observation. Level 3 included patients that needs to be treated on priority in ED clinics. While, Level 4 included non-emergency patients. The study protocol was approved by the Ethics Committee Board of Fujian Provincial Hospital and the requirement of written informed consent from the study patients was waived.

Patient population

Between January 2017 and December 2019, a total of 3146 patients with chest pain visited the ED’s first aid room and were triaged to Level 1 and 2. Patients aged 18 years or more, complaining of acute non-traumatic chest pain, and suspected to be presenting with ACS, as determined by physicians based on their clinical judgment, were included in the study. While, patients diagnosed with ACS prior to ED admission; cardiogenic chest pain such as aortic dissection, pericarditis, cardiomyopathy; non-cardiogenic chest pain caused by gastroesophageal reflux, pulmonary embolism, ruptured esophagus, tension pneumothorax, rheumatic heart diseases, cancer, etc.; and those with missing data were excluded from the study. The diagnosis of cardiogenic and non-cardiogenic chest pain was achieved according to the Chinese Expert Consensus on Standardized Evaluation and Diagnosis of Chest Pain [25]. The patients with occurrence of critical care outcomes during ED treatment were included in the Case group. While, patients without critical outcomes during ED treatment were randomly included in the Control group. In order to satisfy the assumptions of algorithm model, the number of patients in Control group were approximately equal to the Case group. The critical care outcomes were defined as either transfer to ICU, cardiac arrest, or death occurring in the ED, which were reviewed manually by scanning the electronic medical records. The patients with non-traumatic chest pain requiring ICU admission was guided by the guidelines issued by the Emergency Medicine Branch of the Chinese Medical Association. As per these guidelines, the patients with non-traumatic chest pain were admitted to the ICU, if they fulfilled at least one of the following seven criteria [26]: altered consciousness, arterial oxygen saturation < 90% or respiratory failure, significant abnormalities in blood pressure (BP), hemodynamics affected by severe arrhythmia, previous Marfan’s syndrome with severe high BP, and breathing difficulties or full chest on the affected side. The cardiac arrest was defined by unresponsiveness, apnea, and the absence of a central palpable pulse due to pulseless ventricular tachycardia (PVT), ventricular fibrillation (VF), pulseless electrical activity (PEA), or asystole [27]. Data prior to cardiac arrest was used in patients who suffered both cardiac arrest and required ICU transfer, or both cardiac arrest and subsequent death during the treatment in ED.

Patient characteristics and data collection

The routinely available information in the ED was used for the critical prediction model. The information included; 1) Demographics details: age, gender, mode of arrival (ambulance use, transfer from other hospital, walk-in, intra-hospital transfer, and others), and admission and discharge time; 2) Risk factors: history of tobacco use in any form, and family history of premature coronary artery disease (CAD), diabetes, hypertension, hyperlipidemias, and obesity; 3) ED presentation: quality, location, and duration of chest pain, time of arrival, height, weight, Killip class, vital signs and mental status at triage, and complications such as acute heart failure; 4) Initial evaluation: ECG findings (including characteristics of ECG, QT interval, QTc interval, the change of ST-segment, non-specific abnormalities, etc.) and laboratory tests [N-terminal pro brain natriuretic peptide (NT-proBNP), cardiac troponin I (cTnI), serum creatinine (SCr), creatine kinase (CK), and creatine kinase-MB (CKMB)]; 5) Medical treatment: current reperfusion therapy [PCI, CABG, or none]. To ensure the quality of the data, the clinical information and outcomes of all the patients were extracted manually from the medical records. Two researchers were independently involved in data collection. While one screened the participants from electronic medical records based on the inclusion and exclusion criteria, another blinded to clinical outcomes, reviewed all available ED records for complete assessment.

Model development and validation

The data was randomly divided into two sets, the training set (70% of the patients) and the testing set (30% of the patients). In training set, LASSO regression was used to effectively select the important predictors and improve the interpretability of the model through shrink regression coefficients toward zero. Multivariate logistic regression analysis was used to generate independent predictors of critical care outcome in patients with chest pain. Finally, based on nomogram, a visualized LASSO prediction model was established. In both the sets, we computed the model performance, as the discrimination, and calculated 1) Area under the receiver-operating-characteristics curve (AUC); 2) Results of confusion matrix (i.e., accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score); and 3) Net benefit through decision curve analysis. The calibration was appraised by the Hosmer-Lemeshow (HL) test. Moreover, to evaluate the superiority of prediction capability of LASSO model, based on the above metrics, we compared it with the reference model i.e., HEART, GRACE, and TIMI score.

Statistical analysis

Normality of the data was test by Kolmogorov-Smirnov test. If the continuous variable were normally distributed, then they were represented as mean ± standard deviation (SD), else they were represented as median [interquartile range (IQR)]. While, categorical variables were presented as frequencies (percentages). Between group comparison of categorical and continuous data was performed by the Student’s t-test or Wilcoxon’s test, and Chi-square or Fisher’s test, respectively. Missing values were imputed by random forest. All statistical analyses were performed with R software (version 3.5.1; http://www.Rproject.org). A P value less than 0.05 was regarded as statistically significant.

Results

Patient characteristics

The study consisted of 219 patients with critical care outcomes in the Case group, and randomly selected 264 stable patients in the Control group. Then, we randomly assigned 338 (70%) patients to the training set, and the remaining 145 (30%) patients to the testing set, as illustrated in Fig. 1. Patient characteristics in the training and testing sets are depicted in Table 1. There was no significant difference between the two sets in terms of any of the characteristics evaluated.

Fig. 1

Flowchart of the patient recruitment

Table 1 Characteristics of Training ang Testing sets

Selection of features for critical care outcomes in patients with chest pain

On the basis of 338 patients in the training set, 40 features were reduced to 14 potential predictors and these features had non-zero coefficients in the LASSO regression model, illustrated in Fig. 2. These features included gender, mode of arrival, smoking, number of risk factors, reperfusion therapy, Killip class, ECG findings, temperature, respiratory rate, systolic BP (SBP), shock index, SCr, CKMB, and BNP.

Fig. 2

Critical care outcomes risk factors selection using the least absolute shrinkage (A) and selection operator (LASSO) binary logistic regression model (B)

Development of a critical care outcome prediction model

Multivariable logistic regression analysis identified model of arrival, reperfusion therapy, Killip class, SBP, SCr, CKMB, and BNP as independent predictors of critical care outcomes in patients with chest pain (Table 2). These seven independent predictors were then used to develop a LASSO regression model, which was presented as a nomogram (Fig. 3). Further, we developed a point score for predicting the outcomes in critically ill patients, as depicted in Table 3. The probability and risk stratification for each score point is depicted in Table 4.

Table 2 Risk factors for critical care outcome among chest pain patients

Fig. 3

Nomogram for prediction of critical care outcomes in patients with chest pain. SCr: serum creatinine, CKMB: creatine kinase-MB, BNP: brain natriuretic peptide

Table 3 Risk score for critical care outcome among chest pain patients

Table 4 Probability of critical care outcome and corresponding risk stratification

Performance of the critical care outcomes prediction model

For the discrimination based on the training set, LASSO regression model achieved a good result with AUC of 0.924 (95%CI: 0.896–0.952), which was superior to HEART, GRACE, and TIMI score with AUC of 0.699 (95%CI: 0.644–0.754), 0.737 (95%CI: 0.684–0.791), and 0.701 (95%CI: 0.646–0.756), respectively, as illustrated in Fig. 4A. Moreover, compared with these three reference models, LASSO regression model demonstrated a higher accuracy, sensitivity, specificity, PPV, NPV, and F1 score, as depicted in Table 5. With regard to the calibration, the HL test yielded non-significant statistic in LASSO regression model (P value = 0.983), which suggested that there was no departure from perfect fit. Similarly, the decision curve analysis (illustrated in Fig. 5A) demonstrated that the net benefit of LASSO regression model surpassed that of the comparison models throughout the threshold range.

Fig. 4

ROC for the critical care outcomes prediction in training set (A) and testing set (B)

Table 5 Performance of LASSO and comparison models in predicting the critical care outcomes in patients with chest pain

Fig. 5

Decision curve analysis for the critical care outcomes prediction in training set (A) and testing set (B)

Validation of the critical care outcomes prediction model

For the discrimination based on the testing set, LASSO regression model with AUC of 0.953 (95%CI: 0.922–0.984) outperformed the reference models HEART, GRACE, and TIMI score with AUC of 0.754 (95%CI: 0.675–0.832), 0.747 (95%CI: 0.664–0.829), and 0.735 (95%CI: 0.655–0.815), respectively, illustrated in Fig. 4B. Consistently, LASSO regression model produced better outcomes regarding the metrics of accuracy, sensitivity, specificity, PPV, NPV, and F1 score, as depicted in Table 5. Good calibration was observed for the probability of critical care outcomes, with HL test reporting a non-significant statistic for our model (P value = 0.854 and 0.737, respectively). Most importantly, the decision curve analysis demonstrated that LASSO regression model has a higher value on clinical application than HEART, GRACE, and TIMI score, as illustrated in Fig. 5B.

Discussion

We applied machine learning approach - LASSO regression model - to predict the likelihood of complications requiring ICU care, cardiac arrest, or mortality amongst the patients with chest pain admitted in ED. The LASSO regression model comprises of 7 clinical features available at initial contact of patient with ED i.e., mode of arrival, reperfusion therapy, Killip class, SBP, SCr, CKMB, and BNP. Compared to the reference models, the LASSO regression model demonstrated a superior performance in predicting critical care outcomes, including improved AUC value and other metrics. Moreover, the decision curve analysis revealed that LASSO regression model yields a larger net benefit—the trade-off between appropriate prediction and over-prediction—throughout the full range of thresholds. The use of this objective risk stratification tool may help the hospitals effectively use the limited ED resources while ensuring that high-risk chest pain patients are taken care of safely.

The reasons for the improved predictive abilities observed with the LASSO regression model are multifactorial. Firstly, the present study employed complete set of information; for instance, demographics, risk factors of ACS, ED presentation, initial laboratory and ECG findings, and medical treatment, all of these could have resulted in an improved predictive ability. Raita et al. [22] proposed an ED triage system to predict critical care outcomes—direct admission to an ICU or in-hospital death—based on the limited set of predictors only collected at ED triage, such as demography, triage vital signs, chief complaints, and patient comorbidities. Compared with this ED triage systems, the LASSO regression model outperformed in terms of the value of AUC, sensitivity, and specificity (0.86, 0.80, and 0.76, respectively). This is attributable, at least in part, to the limited set of predictors employed. Nevertheless, the differential study purposes and participants make it impossible to compare both the systems. The risk score generated by Raita et al. was proposed for emergency triage and rapid identification of the priority patients. However, it could compromise the accuracy. Contrarily, the LASSO regression model is proposed to be used in emergency room rather than at triage. We ensured the patient safety first, even though time consuming. The participants enrolled in both the studies make it impossible to compare both the systems, and the risk score proposed for emergency patients cannot be directly generalized to patients with chest pain. Moreover, the risk score, in the present study, had some indicators that overlapped with ACTION ICU score [8], which included SCr, SBP, and reperfusion therapy. The findings of some previous studies are inconsistent with that of the present study, notably, cTnI and CKMB have been used for several decades in diagnosing patients with chest pain and to stratify them into those with myocardial and non-myocardial infarction [28]. Estimation of serum troponin is the golden standard of evaluating the cardiac markers [26], however in the present study, cTnI failed to show a statistically significant difference in recognizing the critical care outcomes, nevertheless serum CKMB was measured. A plausible explanation for this inconsistency may be due to the differences in testing point between these two cardiac markers, where cTnI was estimated at an emergency triage immediately on arrival at the ED, and the CKMB was measured before the occurrence of critical care outcomes during the emergency room treatment. Thus, there was an obvious time difference in their estimation. Moreover, in ACTION ICU score, prior revascularization was associated with lower likelihood of developing in-hospital complications requiring ICU care. While, conversely, in the present study, it served as a risk factor because reperfusion therapy was defined as a current therapy, and may have a higher likelihood of presenting to the ICU post-operation.

Secondly, an alternative approach to enhance the predictive ability is to utilize advanced ML algorithm, which is capable of handling high-order interactions amongst the enormous predictors, remarkably, combing them in non-linear highly interactive ways [29]. Recently, ML approaches have opened up vast possibilities in emergency medicine—e.g., cardiac complications in patients with acute chest pain [30], cardiac arrest in ED patients [19, 20], and an ED triage tool for all adults patients [22] or children [21]. The present study confirms that ML models can attain a superior predictive ability for critical care outcomes in patients with acute chest pain. While over-fitting, often generates spurious correlations in the data, we were seriously concerned and thus, adopted multiple rigorous approaches to mitigate, regularize, and validate the independent cohort. Consequently, the performance of the validation model exceeded that of the development model. Decision curve analysis was used to evaluate the feasibility of the proposed model, the result demonstrated that we need to have an optimum balance between under-prediction and over-prediction. The ML model, used in the present study, enables correct identification of the critically ill patients, which might be inappropriately under-triaged into the lower-risk by the HEART, GRACE, TIMI score. Similarly, our model could rule out stable patient which would be over-triaged into high risk patient with those three reference scores, and thus may require additional resources. This finding supports the generalizability of the LASSO regression model. Moreover, this model can be employed directly during the bedside rounds.

Finally, due to the wide usage of HEART, GRACE, and TIMI scores and specifically recognized ability to stratify the patients with cardiovascular system, it may be the best reference models for the present study. We found that our LASSO regression model outperformed to these three well-known scores, and all of these showed a low discrimination index (AUC). Moreover, HEART score was slightly higher than the other two clinical risk scores in detecting the critically ill patients with chest pain in the testing set. To our knowledge, this is the first study to use HEART, GRACE, and TIMI scores in predicting critical care outcomes with the patient of chest pain. We speculate that the poor performance of these three risk scores might be attributed to the indicator of ECG test result they included, such as ST-elevation or ST-depression. There is a time window of clinical deterioration in patients with chest pain. Riley et al. found that 11% of patients with ST-elevation myocardial infarction (STEMI) had an initial non-diagnostic ECG with median time of 72 min between symptom onset and first medical contact, while 72.4% of these patients had an elevation of ST segment after 90 min [31]. Likewise, in the present study, these three scores was calculated only by applying the clinical data at triage, where some predictors remained normal. Several studies [15, 16, 18, 32] have previously compared the prognostic value of different risk scores for predicting MACE among chest pain patients, which consistently favored HEART score over other clinical risk scores in stratifying high-risk patients regarding the onset of MACE. However, in the present study, this prominence was not obviously evident due to the heterogeneity of the study populations, because more severe patients (high risk patients) with Level 1 and Level 2 chest pain who received treatment in the first aid room were included. A meta-analysis demonstrated that the discrimination accuracy of HEART score for the low-risk group was significantly higher than that of the high-risk group [17]. Irrespective of cause, our ML model resulted in an outstanding performance in terms of the AUC value, results of confusion matrix, and the assessment of clinical use.

The current study had several potential limitations. Firstly, this was a single-center study performed at a tertiary provincial emergency center in China, thus the institutional factors and potential selection bias might have resulted in findings that are less generalizable. Secondly, it is a fundamental case-control study of medical history data, with an intrinsic limitation in precision that entails. Nevertheless, most of the events contemplated are concrete and were truly recorded in the electronic medical records system. Thirdly, though the proposed model has demonstrated perfect performance in internal validity, there is a need for external validation of the scores for routine clinical use. Fourthly, we did not capture other clinical features, such as heart rate variability (HRV). The HRV has been regarded as a promising predictor that is recognized to have a significant relationship between the autonomic nervous system and cardiovascular mortality [30, 33,34,35]. Due to the complicated estimation, time consuming procedure, and unsuitability with non-sinus rhythm [35], HRV has not been widely used clinically, especially in the developing country, and thus, was not included in the present study. Finally, the indication and clinical threshold of ICU admission vary depending on the local healthcare resource, such as ICU transfer criteria, ICU bed availability, and the ratio of nurse/patient and nurse/doctor.

Conclusions

To conclude, based on the ML model, we proposed a visualized LASSO regression model using 7 routinely captured clinical features. Compared to well-known clinical risk score—HEART, GRACE and TIMI score, our model had a superior performance in predicting the critical care outcomes in patients with chest pain. Moreover, the model minimized the potential over-predicted and under-predicted critical care outcomes that could result in excessive resource allocation to low-risk patients and insufficient treatment of high-risk patients. While external validation remains essential, the present study may pave the way for the application of ML-based predication models in critically ill patient with chest pain, as a decision-making technological tool.