Identification of avoidable patients at triage in a Paediatric Emergency Department: a decision support system using predictive analytics

Abstract

Background

Crowding has been a longstanding issue in emergency departments. To address this, a fast-track system for avoidable patients is being implemented in the Paediatric Emergency Department where our study is conducted. Our goal is to develop an optimized Decision Support System that helps in directing patients to this fast track. We evaluated various Machine Learning models, focusing on a balance between complexity, predictive performance, and interpretability.

Methods

This is a retrospective study considering all visits to a university-affiliated metropolitan hospital’s PED between 2014 and 2019. Using information available at the time of triage, we trained several models to predict whether a visit is avoidable and should be directed to a fast-track area.

Results

A total of 507,708 visits to the PED were used in the training and testing of the models. Regarding the outcome, 41.6% of the visits were considered avoidable. Except for the classification made by triage rules, i.e. considering levels 1,2, and 3 as non-avoidable and 4 and 5 as avoidable, all models had similar results in model’s evaluation metrics, e.g. Area Under the Curve ranging from 74% to 80%.

Conclusions

Regarding predictive performance, the pruned decision tree had evaluation metrics results that were comparable to the other ML models. Furthermore, it offers a low complexity and easy to implement solution. When considering interpretability, a paramount requisite in healthcare since it relates to the trustworthiness and transparency of the system, the pruned decision tree excels.

Overall, this paper contributes to the growing body of research on the use of machine learning in healthcare. It highlights practical benefits for patients and healthcare systems of the use ML-based DSS in emergency medicine. Moreover, the obtained results can potentially help to design patients’ flow management strategies in PED settings, which has been sought as a solution for addressing the long-standing problem of overcrowding.

Introduction

Emergency Department (ED) crowding occurs when demands are greater than the hospitals’ capacity to ensure timely care in the ED. This is a multifactorial problem with multiple solutions. These problems can be tackled by influencing demand e.g., implementing gatekeeping policies, or optimizing the service provided. Furthermore, ED overcrowding and high patient volumes can result in delays in care, suboptimal treatment decisions, and increased risk of adverse events, including mortality, both in paediatric and adult settings [1,2,3]. To address these challenges, various approaches have been proposed. One of the most studied is the alteration of patient flow i.e., the implementation of fast tracks, based on the severity of the patients’ condition. This approach have shown to be able to improve both efficiency and outcomes, as well as reducing waiting times and overcrowding [4, 5].

Accessing the patients’ severity i.e. the identification of clinically divertible attendances or clinically unnecessary attendances patients [6], to be steered to fast tracks, has been made mostly using triage levels [7,8,9,10] or empirical rules [11]. The validity and accuracy of the identification of these avoidable visits is paramount [12], since it evolves not only patient flow efficiency but also patient safety. On the other hand, the emergence of robust machine learning (ML) algorithms has shown potential to improve predictive ability of various outcomes [13,14,15,16], which in turn could be leveraged to aid in the identification of these avoidable visits [16, 17].

In the hospital this study is being conducted, the Paediatric Emergency Department (PED) is undergoing restructuring, which includes the introduction of a fast-track system. Consequently, there’s a need to develop an algorithm to identify patients who are suitable for this expedited care pathway. A system that identifies these patients accurately has the potential to reduce the amount of time patients spend in the ED, reducing departmental crowding and ultimately support better patient outcomes. It will also, in all likelihood, help the reduction of ED overcrowding and facilitate a more effective allocation of healthcare resources.

Hence, we aim to create a data-driven, optimized DSS to aid in the selection of patients deemed avoidable and redirect them for a fast track. To achieve this, we compared different ML models and approaches, considering the balance between implementation complexity, predictive performance, and interpretability.

Methods

Study design, setting and participants

This study was an observational and retrospective research, conducted in a university-affiliated metropolitan hospital's PED. The hospital serves a population of about 800,000 and receives an average of 76,000 visits annually from an estimated 137,016 children or adolescents aged 0 to 17 years [18].

In the PED, there are 4–5 physicians and 7–8 nurses, working in 12-h shifts to ensure 24/7 coverage.

In this study, all presentations made to the hospital’s PED (i.e., from 0 to 17 years old) in a 4-year period (between 01/Jan/2016, and 31/Dec/2019) were considered.

The PED nursing team triages visitors according to the Canadian Triage and Acuity Scale paediatric guidelines (PaedCTAS), which is structured around evaluating physiological factors, including appearance, neurological status, respiratory rate, heart rate, and perfusion, alongside presenting symptoms to determine triage levels. Similar to the adult version of the Canadian Triage and Acuity Scale (CTAS), the PaedCTAS delineates five levels of triage i.e. Level 1(Red)—“Resuscitation”, Level 2 (Orange)—“Emergent”, Level 3 (Yellow)—“Urgent”, Level 4 (Green)—“Less Urgent” and Level 5 (Blue)—“Non Urgent”. These levels reflect the severity and urgency of the patient's condition, target times for medical assessment and intervention, and provide examples of typical clinical presentations and critical diagnoses [19].

This paper follows the structure presented in the RECORD statement i.e. The REporting of studies Conducted using Observational Routinely-collected health Data [20].

PED restructuring

The hospital's PED is undergoing a significant restructuring.. Along with this restructuring, a fast track for avoidable patients is to be implemented. A schema of the fast-track configuration is presented in Fig. 1.

Fig. 1

Schema of the restructuring of the Paediatric Emergency department regarding avoidable patient flow

Data collection

All patient data is registered using a proprietary information system called JOne, where events are logged e.g. attending medical staff, diagnoses and cause of admission. All sociodemographic information and triage procedures are also registered in this information system.

All data access permissions, i.e., from the hospital board of directors, hospital epidemiology centre, information access officer and ethical committee were granted for this study. [FMUP 180/18].

Outcome

The dependent variable was defined by the conjunction of several PED markers, resulting in a restrictive definition of avoidable. A PED visit is considered avoidable if the patient is discharged home and no diagnostic tests (i.e., blood tests and radiology exams), procedures or medications were required during the stay. The patients were also not asked to stay in the ED, for the physician to better assess the condition’s evolution. In summary, an avoidable visit is done by a patient that only has contact with the physician and is discharged home. The bullet list below summarizes the approach taken in the construction of the outcome to be predicted.

• Avoidable visit;

  • ◦ The patient was not medicated

  • ◦ The patient did not undergo any radiologic exams

  • ◦ The patient did not undergo any blood analysis

  • ◦ The patient did not stay for observations

  • ◦ The patient was discharged home

Predictors and feature selection

A list of variables known at the time of the patient’s triage were used as predictors. Feature selection was done on a model-by-model basis, since associations between variables, redundant variables and low variance variables are handled differently depending on the model.

• Age

• Sex

• Season – Season of the year of patient’s arrival to the PED

• Month – Month of the year of patient’s arrival to the PED

• Day of week –weekday of the patient’s arrival to the PED

• Hour of day – Time of day of patient’s arrival to the PED, in. hourly slots

• Pretriage discriminator group –Paediatric Assessment Triangle group selection as described in PaedCTAS [19]

• Pretriage discriminator –Paediatric Assessment Triangle selection as described in PaedCTAS [19]

• Main complaint group—Triage complaint group as described in PaedCTAS [19]

• Main complaint discriminator – Triage complaint as described in PaedCTAS [19]

• Residence municipality – patient’s residence, municipalities outside the catchment area were residual and were grouped as other

• Triage level – PaedCTAS level of triage

• Referral – A referral patient was defined as ‘not walk-in’ patient e.g. referral from PCP, private clinic or other hospitals

• Made return visit X hours prior- Patient made at least one visit to the PED X (i.e.12, 24, 48, 72) hours prior to current visit i.e. the current visit is a return visit

• Visit by frequent attender – Visit made by frequent attender i.e. > 4 visits per year

Machine learning models

Several ML models were created to evaluate the appropriateness for implementation in this context. These models ranged from simple rules, based on triage level (i.e. visits triaged levels 1, 2 and 3 are directed to regular emergency department patient flow while visits triaged levels 4 and 5 are directed to fast track), to a neural network. The appropriateness involved three dimensions: (1) complexity, some models could be implemented as simple rules, others need to be integrated into the hospital’s information system; (2) interpretability, can the reasoning behind the decision be understood i.e., glass box model or not i.e., black box, and (3) predictive performance. All the models used in this study and their classifications are enumerated in Table 1.

Table 1 List of classifications models used to predict if a visit to the Paediatrics emergency department is avoidable and should be directed the fast-track area

The ML models tested have very different characteristics and some have parameters that are set in advance and control various aspects of the training process itself i.e. hyperparameters. These hyperparameters can be changed, and the performance of the models evaluated in a process called hyperparameter tunning. As an example, and considering decision trees, cost complexity is used for pruning the tree to avoid overfitting. It adjusts the trade-off between accuracy (and possible overfitting) and tree simplicity.

The default threshold of 0.5 for binary classification was kept for all the models.

Data preparation and model training

The dataset was randomly split 20% for testing and 80% for training with stratification for the outcome. On the training set, it was used a 2-fold cross validation with stratification for the outcome, repeated 20 times.

All categorical missing values were imputed the category “unknown”, there were no missing numerical data. Only the variables “Pretriage discriminator” and “Pretriage discriminator group” had an expected, and relevant missing count (96.7%). These variables are only filled in particular circumstances during the triage algorithm. All other categorical variables had negligible (> 0.5%) of missing count.

For tensor flow model and the XGboost model, dummy variables were created i.e., one binary numeric variable was created for each category. Furthermore, the variable “Triage maincomplaint discriminator” was removed from the simple tree model because it has too many categories (217) and would be unusable in the paper-based approach. To mitigate complexly, the tree depth maximum was also set to 5 in hyper parameter tuning phase. The variable “Triage maincomplaint discriminator” was also removed from the logistic regression model, as some levels had only a small number of observations and the information was already aggregated in the variable “Main complaint group”.

All the data analysis was performed using R version 4.2.2 (2022–10-31) [23]. The integrated development environment (IDE) used was RStudio Version 2022.12.0 + 353 [24]. The ecosystem of packages “tidymodels” was used. The specific packages used for training the ML models are underlined in Table 1.

Models’ evaluation

The evaluation metrics used in our study to compare the performance of different machine learning methods are enumerated below. Accuracy (accuracy), Negative Predictive Value (npv), Positive Predictive Value (ppv), Sensitivity (sens) Specificity (spec), Kappa (kap), Area Under Curve (roc_auc) and F-measure (F_meas) that combines ppv and sensitivity, providing a single score that reflects both aspects of a model's performance. Furthermore, False Positives (FP) i.e., non-avoidable visits classified as avoidable, False Negatives (FN) i.e. avoidable visits classified as non-avoidable, True Positives i.e. visits classified correctly as avoidable and True Negatives (TN) i.e. visits classified correctly non-avoidable were also computed for each model.

Results

The dataset utilized for training and testing the models comprised a total of 507,708 visits to the pediatric emergency department. Of these visits 17.4% were referrals, and 4.4% resulted in hospital admissions. Females accounted for 46.8% of the visits. The average age of patients was 7 years, with a standard deviation of 5.5 years. The mean length of stay was 102.4 min, with a standard deviation of 154.5 min. Regarding the outcome, 41.6% of the visits were considered avoidable. Concerning triage levels, 0.2% were triaged level 1, 5.6% level 2, 39.4% level 3, 50.4% level 4 and 4.5% level 5.

Models’ performance

Considering the metrics that evaluate the models globally i.e. accuracy, f_meas, kap and roc_auc, all ML models outperformed the classification made by triage rules i.e. visits triaged levels 1 (red), 2 (orange) and 3 (yellow) were considered non-avoidable and visits triaged levels 4 (green) and 5 (blue) were considered avoidable. Except for the classification made by triage rules, the models had similar performance, namely accuracy, ranging from 70 to 72% and AUCs ranging from 74 to 80%. However, regarding measures that evaluate the specific performance i.e., NPV, PPV, sensitivity and specificity, there is greater variation Moreover, the triage rules model was also outperformed by the all ML models in terms of measures addressing specific performance, except for sensitivity.

Predictions on test data

For a more pragmatic and in-depth analysis and drilling down from Figs. 2 and 3 presents a confusion matrix’s inspired plot, where the model’s classifications are compared. This visualization enables us to calculate, for a particular number of visits, how many are correctly classified and misclassified, for any given model. It is important to highlight the low FP proportion of all ML models and the high TN proportion, with relatively small differences between them. On the other hand, the classification made by triage rules had the highest FP proportion and the lowest TN proportion. The FN rate and the TP rate were relatively constant across all models.

Fig. 2

Bar plot with error bars, comparing the performance of the predictive models used to classify if visit to the Paediatrics emergency department is avoidable, and should be directed the fast-track area, by metric. Results were obtained from the twofold cross validation with stratification for the outcome, repeated 20 times

Fig. 3

Percentage of visits, for each model, incorrectly assigned to the fast-track i.e., non-avoidable classified as avoidable (FP), incorrectly assigned to normal flow i.e. avoidable classified as non-avoidable (FN) and percentage of visits classified correctly, either avoidable (TP) or non-avoidable (TN). Data was obtained from the test dataset when the models were fitted

Simple classification tree

When constructing the model for the simple classification tree, the hyperparameter tunning was limited to a tree depth of 5. Fig. 4 shows that above a tree depth of 4 there is no significant improvement in overall performance i.e., accuracy and AUC. Furthermore, when the value of cost complexity decreases, sensitivity increases.

Fig. 4

Plots with the results from a grid hyperparameter tunning for the simple decision tree. Each plot refers to a specific tree depth. Metric’s mean is the result of a twofold cross validation repeated 20 times

Figure 5 presents a decision tree with the hyperparameters set to: tree depth of 4 and a cost complexity of 1e-4. This combination was chosen for the best balance between sensitivity and specificity, not having a significant impact on overall performance i.e., accuracy and AUC. This particular tree only used the triage’s “Main complaint group” and triage level.

Fig. 5

Decision tree with the hyperparameters set to tree depth of 4 and a cost complexity of 1e-4, for the classification of visits as avoidable (Yes) or non-avoidable (No), and redirection to a fast track or regular flow respectively, in the Paediatric emergency depart. This tree was created from the test dataset

Discussion

The major objective of this study was to evaluate several ML models to be implemented in the PED and aid in the decision if a visit to the Paediatrics emergency department is avoidable and should be directed to the fast-track area or is non-avoidable and can stay in the regular flow.

To the best of our knowledge, the assignment of visits to a low acuity fast track is mostly done by the assigned triage levels and there are no studies with a pragmatic focus on implementation with data-driven approaches. [7,8,9,10, 16, 25, 26].

Our approach was to leverage the power of ML to aid in the assignment for the fast-track. First, creating an outcome based on resource utilization and PED discharge destination. And afterwards trying to predict it using information known at the time of triage.

Summary of main findings

Regarding overall performance metrics, all ML models had similar performance and outperformed the classification made by triage rules. Drilling down, errors made by triage rules were mainly false positives, i.e. non avoidable patients sent to the fast track.

The pruned decision tree performed only slightly worse in the overall metrics than all the other ML models and their errors in classification went in the same direction as the other more complex models, as can be seen when analysing sensitivity, specificity or the confusion matrix.

Results contextualization

This study's results regarding AUC were slightly higher than those found by Chang et al. who made a similar study using ML to identify low-severity patients. However, the definition of candidates for fast track was the time interval between the triage registry and being discharged within less than 4 h. Therefore, depending on the setting, this could be a problem, since the length of stay is greatly influenced by the triage level and the hour of the day. Hence, we think that this study's approach, only considering resource usage and discharge destination, appears less prone to confounders [27].

In a study by Kwon et al. [28] where the aim was to identify high-risk patients at the time of triage, the ML algorithm predicted in-hospital mortality, critical care, and hospitalization more accurately than existing triage systems. When considering only hospitalization, the results are very similar to our study [28]. Despite the different outcomes, these results support the superiority of ML models over triage system’s classification to predict outcomes in the ED. This is further reinforced and expanded by the results of a systematic review by Kareen et al. where it was found that in an ED setting, ML models outperformed usual care in most diagnostic and prognostic predictions across all studies [16].

It is essential to consider the FP values for each model, i.e., visits that the models classified as avoidable that really were not avoidable. While false negative (FN) proportion, i.e., visits that the models classified as non-avoidable that were avoidable, are directed to the previous patient flow, a high FP proportion might be a reason for concern since they are non-avoidable visits directed to the fast track. In this regard, the split made by triage rules is a reason for worry, since its FP proportion doubles the highest ML model. This highlights the necessity to include all relevant stakeholders (e.g. physicians, nurses, administrative personnel and patients) in the development of the DSS and is vital to its successful implementation, as only these stakeholders know the particularities of the context where the DSS is to be deployed. The ML models should be evaluated considering not only the performance metrics but also potential clinical consequences and impact on patient outcomes [16, 29].

Implications for policy and practice

To discuss the choice of the model to be implemented is necessary to return to the three evaluation dimensions stated in the introduction: complexity, predictive performance, and interpretability. The pruned decision tree is very simple to implement. Based on two or three simple rules, a nurse at the end of triage could redirect the patient with minimal increased logistic burden. As a consequence of its simplicity, the time to implementation and cost is negligible. Regarding the performance assigning patients to a fast track based on triage levels is clearly not the best solution, given the high proportion of FP. Despite not being the best model, the pruned decision tree is outperformed by other more complex models but only by a few percentage points. The last dimension is extremely important and goes beyond interpretation, it relates the understanding to how the decision was made, and if the explanation is satisfactory, it builds trust [30]. And a trustworthy and explainable system has been a stakeholder requisite, especially in healthcare [31, 32]. In this regard, rules extracted from a low-depth decision tree excels.

The choice of the model was clear. The pruned decision tree could be implemented immediately, and the patients and staff could immediately reap the rewards of an improved PED workflow. The resources committed to the implementation are low, hence can be easily replaced if a better alternative is developed.

Limitations

The definition of avoidable visit was chosen among others [6, 33, 34] for being triage system agnostic, better reflecting the visit’s lack of necessity for the hospital’s resources. Nevertheless, the multiple definitions used in the field make comparisons less accurate.

There are few clinical parameters available for model building. Nevertheless, if more variables were to be collected and used, the model's performance could never worsen.

The data gathered had the original purpose of providing care to the patients in the PED, therefore subjected to the bias of any observational study based on routinely collected data, e.g. information system downtime and the inability to control how the variables are collected.

The deployment phase will present challenges and model’s suggestions might have to be adapted the day-to-day operations and decision-making processes in clinical settings.

External validity of the models was not tested. However, given the intended use of the models, it's crucial to consider both the target population and the setting. Thus, if this study were to be conducted in different settings or with different populations, the models would need to be retrained with the data available [35].

Conclusions

This study demonstrates the substantial potential of ML models to enhance decision-making processes in Emergency Departments, regarding the assignment of patients to appropriate care paths. Our findings underscore the superior performance of ML models over more traditional methods in determining the patient flow. While all tested ML models performed well, the pruned decision tree model emerged as a practical choice due to its simplicity, ease of implementation, and relatively high accuracy. Moreover, this model supports the need for interpretable and trustworthy systems in healthcare, as it allows healthcare providers to understand and trust the basis of its predictions. Moving forward, the integration of ML into clinical settings should continue to focus on balancing complexity, predictive performance, and interpretability, ensuring that such tools are not only technically effective but also align with the practical realities and ethical considerations of medical practice. Finally, the obtained results can potentially help to design patients’ flow management strategies in PED settings, which has been sought as a solution for addressing the long-standing problem of overcrowding.

More data-driven approaches, where the patient and healthcare professionals are put first and technology serves an instrumental role in solving the problem, are necessary in this age of AI hype. Small, targeted interventions to solve real-world problems with real-world data are paramount to the future of healthcare.