The Sydney Triage to Admission Risk Tool (START) to predict Emergency Department Disposition: A derivation and internal validation study using retrospective state-wide data from New South Wales, Australia
BMC Emergency Medicine volume 16, Article number: 46 (2016)
Disposition decisions are critical to the functioning of Emergency Departments. The objectives of the present study were to derive and internally validate a prediction model for inpatient admission from the Emergency Department to assist with triage, patient flow and clinical decision making.
This was a retrospective analysis of State-wide Emergency Department data in New South Wales, Australia. Adult patients (age ≥ 16 years) were included if they presented to a Level five or six (tertiary level) Emergency Department in New South Wales, Australia between 2013 and 2014. The outcome of interest was in-patient admission from the Emergency Department. This included all admissions to short stay and medical assessment units and being transferred out to another hospital. Analyses were performed using logistic regression. Discrimination was assessed using area under curve and derived risk scores were plotted to assess calibration.
1,721,294 presentations from twenty three Level five or six hospitals were analysed. Of these 49.38% were male and the mean (sd) age was 49.85 years (22.13). Level 6 hospitals accounted for 47.70% of cases and 40.74% of cases were classified as an in-patient admission based on their mode of separation. The final multivariable model including age, arrival by ambulance, triage category, previous admission and presenting problem had an AUC of 0.82 (95% CI 0.81, 0.82).
By deriving and internally validating a risk score model to predict the need for in-patient admission based on basic demographic and triage characteristics, patient flow in ED, clinical decision making and overall quality of care may be improved. Further studies are now required to establish clinical effectiveness of this risk score model.
One of the most important aspects of treating patients in Emergency Departments (ED) is deciding whether a patient is safe for discharge or requires in-patient admission for further treatment and stabilisation . These are termed disposition decisions and they involve the complex interaction of clinical factors such as diagnoses, severity and response to treatment, as well as social and clinician factors. It has been shown that most experienced ED clinicians use clinical judgement to make disposition decisions, however in ED, these are often made in the context of various time pressures, limited clinical information and ED overcrowding which can lead to clinical errors [1, 2].
Clinical decisions can also take time, particularly when there are multiple patients with varying acuity, requiring multiple and simultaneous decisions. A study of process factors in ED demonstrated that discharge and in-patient admission disposition decisions take on average around 130 min and 200 min from ED arrival time respectively . The need for timely accurate and safe decision making in ED has grown more urgent with the increased demand for Emergency Department services, the ageing population, limited in-patient bed capacity and time based ED performance targets [3–5]. Overcrowding in EDs has become a major public health concern in Australia and around the world. One innovative solution involves the use of data analytics and decision support systems to assist clinicians at the point of triage and bedside. A recent study of four hospitals, including two Veteran’s Affairs hospitals in the US developed a prediction model for admission using urgency categories, presenting problem categories and age, with moderately high accuracy (Area Under Receiver Operator Characteristic Curve between 0.80 to 0.89) .
We sought to derive and internally validate a similar model using State-wide Emergency data in Australia, which will ultimately be used to develop and implement a risk score base decision support tool for disposition at point of triage. The objective was to determine if similar measures of overall model accuracy could be obtained using a more generalised dataset containing many ED’s. The driver of the project is to translate data based research into decision support tools that not only assist with process efficiency but also improve the quality of clinical decision making and potentially facilitate shared decision making between clinicians and patients. This may be translated into tools that improve streaming to medical assessment or short stay units, assist clinicians in prioritising the clinical workup for ED patients that have a high probability of being admitted or discharged.
This was a retrospective data analysis of State-wide Emergency Department data and undertaken as part of the Demand for Emergency Services Trend in Years 2010–15 (DESTINY) study .
New South Wales is the most populous state in Australia with a population of around 7.5 million people and a land area of 809, 000 km2 .
The Emergency Department Data Collection (EDDC) Registry routinely collects patient level data on presentations to all designated Emergency Departments in New South Wales (NSW). Data collection includes, referral source (self-referred, General Practice, Specialist, Nursing Home), mode of arrival (self-referral, Ambulance), hospital facility, triage category (Australasian Triage Scale) , presenting problem, mode of separation (admitted to hospital, discharged or died). Presenting problems allocated by triage nurses at the point of patient arrival to ED were categorised into broad clinical groups and described elsewhere (see Table 1) . For example, neurological complaints included headache, dizziness, weakness and ataxia), respiratory complaints included shortness of breath, cough and wheeze and cardiovascular complaints included chest pain and palpitations. The time that a patient was triaged was used as the ED arrival time and categorised using commonly accepted ED clinician shift times (0800–1759, 1800–2359 and 0000–0759).
Hospital facilities were classified according to current Ministry of Health definitions for designation of Emergency Departments based on case-mix, staffing and specialist facilities within each hospital . In brief, these range from Level six centres comprising tertiary level teaching hospital that are Major Trauma Centres (including two specialist paediatric centres), Level five centres tertiary level non trauma centres, Level four centres which are mainly Metropolitan District level hospitals, Level three centres which are smaller district and general hospitals, and level two and one centres which comprise smaller rural multi-purpose and urgent care centres. Estimated Residential Population by age and sex were obtained from the Australian Bureau of Statistics .
Adult patients (age ≥ 16 years) were included if they presented to a Level 5 or 6 Emergency Department between 2013 and 2014. Patients who were dead on arrival or were planned representations were excluded as were patients who did not wait for triage and had presenting problem field entries that were either missing or uninterpretable.
The outcome of interest was in-patient admission from the Emergency Department. This included all admissions to short stay and medical assessment units and being transferred out to another hospital.
Descriptive statistics were used to compare univariate predictors of in-patient admission. The dataset was then randomly split into derivation and validation datasets. Logistic regression with stepwise selection was used to determine predictors in the derivation dataset which was then tested on the validation dataset to determine Area Under Curve (AUC) of Receiver Operator Characteristic (ROC) curves. Reference values for variables were assigned through investigator consensus. Risk scores were assigned based on variable coefficient values and predicted versus observed risk of admission for each score were plotted to obtain calibration curves. The procedure was then repeated using least absolute shrinkage and selection operator (LASSO) regression as an alternative to variable selection.
Access to data was approved by the NSW Population Health Services Research Ethics Review Committee.
A total of 1,773,550 ED presentations were identified of which 52,256 cases (2.95%) were excluded. Of the excluded cases, 26,915 had missing presenting problems, 24,564 (1.41%) had presenting problems that were uncodeable and 777 cases did not wait for triage, leaving 1,721,294 presentations from twenty three level 5 or 6 hospitals. Of these 49.38% were male and the mean (sd) age was 49.85 years (22.13). Level 6 hospitals accounted for 47.70% of cases and 40.74% of cases were classified as an in-patient admission based on their mode of separation.
Univariable and multivariable predictors of in-patient admission
The dataset was randomly allocated so that 860,832 cases (50.01%) were assigned to the derivation dataset. Table one compares the baseline characteristics of admitted and non-admitted ED patients in the derivation dataset. Patients requiring admission were older with higher triage acuity scores, and associated with cardiovascular, respiratory febrile illness and other general medical presenting problems and had a previous in-patient admission within 30 days of current ED presentation.
After excluding non-significant variables in univariate analysis, the final multivariable model including age, arrival by ambulance triage category previous admission and presenting problem (Table 2) had a AUC of 0.82 (95% CI 0.81, 0.82) and with a Hosmer-Lemeshow test statistic p < 0.001 for calibration. When this was repeated using LASSO selection, the AUC was 0.81 (95% CI 0.814, 0.816).
Figure 1 shows the range of risk scores possible with mean admission rates. Deciles of risk score ranges with corresponding mean predicted probabilities of in-patient admission were as follows: Risk score <1 (3%), risk score 1–10 (14%), risk score 11–20 (47%), 20–30 (81%), 30–40 (96%), >40 (99%). These were plotted on a calibration curve is shown in Fig. 2. Over high risk score ranges (risk score >20) the positive predictive value was 86.8%, negative predictive value of 64.25% for in-patient admission. For low risk scores (risk score <10), the positive predictive value for discharge from ED was 89.85% and the negative predictive value was 46.28%. The optimum point on the ROC curve corresponded to a sensitivity of 88% and a specificity of 67% corresponding to a risk score of 13.
This study demonstrates that disposition prediction in ED can be made with reasonable accuracy using only initial presenting problem together with known variables such as age, mode and time of arrival. The risk scores can be summed to obtain an estimate of the risk of inpatient admission or discharge from the emergency department. The authors have named this model the “Sydney Triage to Admission Risk Tool” (START) and is the first such study reported in Australia. The overall accuracy of the model was around 82% with an AUC comparable to previously published clinical risk scores such as the Pneumonia Severity Index and previous data analysis studies of ED disposition from US hospitals [6, 11]. The Hosmer-Lemeshow test statistic suggested that calibration was suboptimal, however this may be a function of the large sample size – indeed the calibration curve showed that any lack of calibration was unlikely to be clinically meaningful.
This study has a number of advantages in that it used state-wide data for all tertiary hospitals across NSW making it applicable across many centres, utilises a small number of variables that can be reliably obtained upon patient arrival by clerical and triage staff in ED. Used in this way, the derived risk score may potentially be used to rapidly identify patients suitable either for immediate in-patient bed allocation (high risk), further assessment in a medical or surgical assessment area (moderate risk), or streamed to a fast track area for those with anticipated discharge (low to negligible risk). In doing so, the time to disposition for patients in ED can be reduced resulting in improved patient flow and efficiency. It may also be used as a decision support at the bedside, perhaps integrated within clinical information systems or mobile devices to assist clinicians managing patients to safely decide where the most suitable destination for the patient should be .
Further studies are now warranted to evaluate this data modelling study and see if model performance translates in actual clinical practice and reduces ED decision times. These are currently being planned in New South Wales and will evaluate the risk score based on the performance reported in this study. Studies are also required to compare model performance against clinician based prediction. One single centre study from Westmead Hospital demonstrated that senior clinicians predicted ED discharge with a positive predictive value of around 0.90 however the positive predictive value for in-patient admission prediction was only around 55% . A study from Queensland used data analytics to predict ED demand based on day of week and previous total ED presentations with the aim of forecasting bed requirements on an administrative level . In contrast this study examined individual factors that predicted disposition with the aim of improving patient flow in ED. Increasing efficiency in ED has been shown to improve quality of care in ED and in-hospital mortality [15, 16]. ED performance is constrained by growing demand, overcrowding and a relatively junior and rotating workforce particularly after hours [15, 16]. These conditions are ideal for risk score based decision tools designed to reduce human factor variation in clinical decision making and to stream patients reliably into different clinical areas.
The study also provides a useful reference point for more advanced data analytic methods such as neural networks to investigate whether model performance can be improved by including facility and specific subgroups of presenting problems. There are around 400 different presenting problems in the dataset, and the major problem with categorising presenting problems as we have done is the overlap between different presenting problems. Shortness of breath for instance can be both a cardiovascular and respiratory complaint. Using more sophisticated machine learning techniques opens the possibility of more refined prediction based on "big data" principles of using specific personal and presentation characteristics, including the use of historical background information, previous diagnoses and presenting problems linked to previous admissions which cannot be analysed using more traditional statistical methods as we have done . Advantages of this approach include the inclusion and analysis of more granular decision making nodes such as the most appropriate clinical service and the most appropriate clinical service location to admit to such as the Intensive Care unit or normal in-patient ward location. It may also also account for the observation that most presenting problem types actually decreased the odds of in-patient admission and this was perhaps due to the combining of fairly disparate presenting problems under broad categories, for instance “cough” and “cyanosis” were both classified under respiratory problems, even though they represent two extremes of respiratory conditions.
It should be noted that the in-patient admission rate of around 40% includes those patients admitted to Emergency Department short stay and observation units, as they are classified under current administrative definitions as an in-patient unit. Notwithstanding this, the in-patient admission rate is still substantially higher than previously reported  in other regions in Australia and requires further investigation. The most likely explanation being that this study included higher level tertiary centres that are more likely to treat more complex patients. Similarly it is unclear why ED arrival time during normal business hours should be more predictive of in-patient admission compared to other hours of the day. It may reflect bed management and hospital specific practices with more referrals from specialty and general practice clinics during the day or it may be a reflection of the types of patients presenting after hours. The impact of in-patient ward availability (access block) on disposition outcomes was also not evaluated in this study. Hospital overcrowding may have a confounding effect on disposition decisions by clinicians, reducing the likelihood of admission in a given day if it is already known that there are no beds in the hospital available. This requires prospective evaluation, which is currently underway.
Other acknowledged limitations include the lack of information of background medical history, which can often be crucial in making disposition decisions. This can potentially be accessed by linkage with admitted patient databases and the use of a cumulative list of previous ED diagnoses within this dataset. It is also difficult to quantify the role of clinical experience and clinical judgement in making disposition decisions and the overall misclassification rate of 18% underscores the importance of those clinical factors, therefore as with any clinical decision rule this model should not be used in isolation but in conjunction with clinical acumen.
Although we have only used data from tertiary hospitals, the majority of hospitals EDs in NSW are smaller non tertiary hospitals. Including such centres may have introduced heterogeneity due to varying triage and admission practices as well as differences in patient presentation patterns. Therefore separate or multilevel analyses are required to incorporate all types of ED’s.
In conclusion, we have derived and internally validated a risk score model to predict the need for in-patient admission based on basic demographic and triage characteristics. This model may be used to facilitate patient flow in ED, standardise clinical decision making and improve overall quality of care. Further translational studies are now warranted to assess model performance in clinical practice and evaluate its impact on patient outcomes.
United States of America
Area Under Curve
Receiver Operator Characteristic
Emergency Department Data Collection Registry
Calder LA, Arnason T, Vaillancourt C, Perry JJ, Stiell IG, Forster AJ. How do emergency physicians make discharge decisions? Emerg Med J. 2015;32(1):9–14.
Casalino E, Choquet C, Bernard J, Debit A, Doumenc B, Berthoumieu A, et al. Predictive variables of an emergency department quality and performance indicator: a 1 year prospective, observational, cohort study evaluating hospital and emergency census variables and emergency department timeinterval measurements. Emerg Med J. 2013;30(8):638–45.
Lowthian JA, Curtis AJ, Jolley DJ, Stoelwinder JU, McNeil JJ, Cameron PA. Demand at the emergency department front door: 10-year trends in presentations. Med J Aust. 2012;196:128–32.
Dinh MM, Bein KJ, Latt M, Chalkley D, Muscatello D. Age before acuity: the drivers of demand for emergency department services in the Greater Sydney Area. Emerg Med J. 2014 Dec 18. pii: emermed-2014-204174. doi:10.1136/emermed-2014-204174. [Epub ahead of print]
Australian Institute of Health and Welfare. Australian hospital statistics 2012–13: emergency department care, Health services series no. 52. Cat. no. HSE 142. Canberra: AIHW; 2013.
Peck JS, Gaehde SA, Nightingale DJ, Gelman DY, Huckins DS, Lemons MF, et al. Generalizability of a simple approach for predicting hospital admission from an emergency department. Acad Emerg Med. 2013;20(11):1156–63.
Dinh MM, Berendsen Russell S, Bein KJ, Chalkley D, Muscatello D, Paoloni R, et al. Understanding drivers of Demand for Emergency Services Trends in Years 2010–2014 in NSW. An initial overview of the DESTINY project. Emerg Med Australas. 2016;28(2):179–86.
Australian Bureau of Statistics. Estimated Residential Population by single year of age, New South Wales. Available at http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/3101.0Jun%202014?OpenDocument Last updated 26 March 2015. Accessed 15 July 2015.
Hodge A, Hugman A, Varndell W, Howes K. A review of the quality assurance processes for the Australasian Triage Scale (ATS) and implications for future practice. Australas Emerg Nurs J. 2013;16(1):21–9. doi:10.1016/j.aenj.2012.12.003. Epub 2013 Feb 8.
NSW Health. Guide to the role delineation of health services. Third edition. 2002. State Health Publication No. SWS 990186 http://www.health.nsw.gov.au/Hospitals/Pages/role-delineation-levels.aspx. Last updated 24 January 2014 Accessed 15 Aug 2015.
Fine MJ, Auble TE, Yealy DM, Hanusa BH, Weissfeld LA, Singer DE, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243–50.
Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerised clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38.
Vaghasiya MR, Murphy M, O’Flynn D, Shetty A. The emergency department prediction of disposition (EPOD) study. Australas Emerg Nurs J. 2014;17(4):161–6. doi:10.1016/j.aenj.2014.07.003. Epub 2014 Aug 8.
Boyle J, Jessup M, Crilly J, Green D, Lind J, Wallis M, et al. Predicting emergency department admissions. Emerg Med J. 2012;29:358–65. doi:10.1136/emj.2010.103531.
Sprivulis PC, Da Silva J, Jacobs IG, Jelinek GA, Frazer ARL. The association between hospital overcrowding and mortality among patients admitted via Western Australian emergency departments. Med J Aust. 2006;184:208–12.
Hoot NR, Aronsky D. Systematic review of emergency department crowding: causes, effects and solutions. Ann Emerg Med. 2008;52:126–36.
Heaton B. Big data How Emergency Managers Can Benefit from Big Data: Large volumes of data sets derived from sophisticated sensors and social media feeds are increasingly being used by government agencies [Internet]. Emergency Management 2013 Jul 23: Available from http://www.emergencymgmt.com/disaster/Emergency-Managers-Big-Data.html. Accessed 10 May 2016.
We would like to acknowledge the NSW Ministry of Health and the Centre for Health Record Linkage (CHeReL) for access and linkage of data.
This project was funded by the New South Wales Agency for Clinical Innovation and the Emergency Care Institute (reference number ACI/D14/2288).
Availability of data and materials
Data used for this study are available by ethics approval and applying to the EDDC http://www.cherel.org.au/data-dictionaries#section9.
MMD conceived the study, was involved with study design, data analysis and manuscript preparation, SBR was involved with study design and manuscript preparation, KJB and KR were involved with data analysis and manuscript preparation, DM was involved with data analysis and manuscript preparation, RP was involved with study design and manuscript preparation, DRC was involved with study design and manuscript preparation, RI and JH were involved with study design, manuscript preparation and expert advice. All authors have read and approved the manuscript.
The authors declare that they have no competing interests.
Consent for publication
The authors provide permission to publish this work
Ethics and consent to participate
The study was approved by the NSW Population Health Services Research Ethics Review Committee. Waiver of consent was approved by this committee as this was a study using a large administrative database (Protocol number HREC/14/CIPHS/38).
About this article
Cite this article
Dinh, M.M., Russell, S.B., Bein, K.J. et al. The Sydney Triage to Admission Risk Tool (START) to predict Emergency Department Disposition: A derivation and internal validation study using retrospective state-wide data from New South Wales, Australia. BMC Emerg Med 16, 46 (2016). https://doi.org/10.1186/s12873-016-0111-4
- Risk score