Skip to main content

Validation of the modified Skåne emergency department assessment of patient load (mSEAL) model for emergency department crowding and comparison with international models; an observational study

A Correction to this article was published on 16 April 2021

This article has been updated

Abstract

Background

Emergency Department crowding is associated with increased morbidity and mortality but no measure of crowding has been validated in Sweden. We have previously derived and internally validated the Skåne Emergency Department Assessment of Patient Load (SEAL) score as a measure of crowding in Emergency Departments (ED) in a large regional healthcare system in Sweden. Due to differences in electronic health records (EHRs) between health care systems in Sweden, all variables in the original SEAL-score could not be measured reliably nationally. We aimed to derive and validate a modified SEAL (mSEAL) model and to compare it with established international measures of crowding.

Methods

This was an observational cross sectional study at four EDs in Sweden. All clinical staff assessed their workload (1–6 where 6 is the highest workload) at 5 timepoints each day. We used linear regression with stepwise backward elimination on the original SEAL dataset to derive and internally validate the mSEAL score against staff workload assessments. We externally validated the mSEAL at four hospitals and compared it with the National Emergency Department Overcrowding Score (NEDOCS), the simplified International Crowding Measure in Emergency Department (sICMED), and Occupancy Rate. Area under the receiver operating curve (AuROC) and coefficient of determination was used to compare crowding models. Crowding was defined as an average workload of 4.5 or higher.

Results

The mSEAL score contains the variables Patient Hours and Time to physician and showed strong correlation with crowding in the derivation (r2 = 0.47), internal validation (r2 = 0.64 and 0.69) and in the external validation (r2 = 0.48 to 0.60). AuROC scores for crowding in the external validation were 0.91, 0.90, 0.97 and 0.80 for mSEAL, Occupancy Rate, NEDOCS and sICMED respectively.

Conclusions

The mSEAL model can measure crowding based on workload in Swedish EDs with good discriminatory capacity and has the potential to systematically evaluate crowding and help policymakers and researchers target its causes and effects. In Swedish EDs, Occupancy Rate and NEDOCS are good alternatives to measure crowding based on workload.

Peer Review reports

Introduction

Emergency Department (ED) crowding is a prevalent problem in many health care systems and associated with increased morbidity, mortality and decreased quality of care [1, 2]. There are several measures of crowding, from simple numeric scores to more complex models which incorporate different measures into a single score [3]. The rationale is that complex models encompass different aspects of crowding and are thought to provide better information compared to single variables such as Occupancy Rate [4, 5]. Crowding in Swedish EDs has previously been limited [6], but has become a problem in recent years, and several research projects on the topic have been initiated [7, 8]. However there is no definition or validated measure of crowding in Sweden which limits the efforts to detect and systematically prevent its negative effects on patient care. Internationally the National Emergency Department Overcrowding Score (NEDOCS) have been derived and validated in the US using staff perception of crowding and generates a score from 1 to 200, with overcrowding defined as a score over 100 [5]. The International Crowding Measure in Emergency Department (ICMED) was derived in the UK against staff perception of crowding and danger. Points are assigned to different aspects of ED care on a scale from 1 to 7, and a score of 3 or higher has a high predictive ability for crowding [4]. Both scores include variables that are not readily available in the ED and there are concerns whether they could be applied to the Swedish healthcare system [9].

Since there are no validated measures of crowding in Sweden, we derived and validated the Skåne Emergency Department Assessment of Patient Load (SEAL) score in a large regional health care system in southern Sweden [9]. We used staff perception of workload as a measure of ED crowding since it is reasonable to assume that a substantial part of the negative effects of crowding are due to high staff workload, and since similar measures have been shown to be reliable in other crowding models like NEDOCS and ICMED [4, 5]. The score is intended for automated calculation based on data from the EHR and internal validation has shown consistent results [9]. So far, the SEAL score has not been externally validated, and its utility has not been compared with existing international crowding models.

Sweden, with a population of 10 million, has a universal publicly funded health care system granting emergency care with a small co-payment at 72 EDs spanning from small rural EDs to large urban academic EDs. The health care is organized into 21 health care systems with different EHRs and ED support systems. Because of differences in the EHRs, all variables in the original SEAL score cannot be measured reliably in many Swedish health care systems at this time. For optimal usability and generalisability, a crowding model needs to include measures readily available in the EHR at many EDs.

Aim

In the present study, we aimed to derive and externally validate a modified SEAL model (mSEAL) to measure ED crowding based on variables reliably available across different EHRs, and to compare it with established international measures of crowding.

Methods

Derivation and validation of mSEAL

We used linear regression analysis with stepwise backward elimination on the original SEAL-dataset [9]. The SEAL model includes the variables Patient Hours, High acuity Patients, Time waiting for a physician and Occupancy rate. In short, Patient hours is the sum of time all patients spent in the ED during the past hour, measured in hours, and has the largest impact on the model score. High Acuity Patients is the proportion of high (highest and second highest out of five categories) acuity patients at the ED. Time waiting for a physician is the time from triage and nursing procedures to first physician contact. Occupancy rate is the number of ED patients divided by the number of ED treatment beds.

Since High Acuity Patients and Time waiting for a physician could not be reliably collected at all EDs, these were left out and exchanged for the new variable time to physician (measuring the time from registration in the ED to first physician contact) to form the mSEAL score. The mSEAL was then internally validated in a separate data collection from two of the five hospitals in the original study, as well as externally validated at four EDs, as described below.

External validation

We collected data at four EDs in Sweden (Table 1) during September to November 2017; Three EDs at academic teaching hospitals and one at a rural community hospital in two different healthcare systems. Workload assessments, rated on a Likert scale from 1 (no workload) to 6 (very high workload), were collected from each included ED at 5 pre-defined time points per day (08:00, 12:00, 16:00, 20:00, 23:00). These time points were selected to represent known periods of both high and low levels of crowding, and were separated in time to reduce the risk of temporal correlation between assessment and scores [10, 11]. All working staff (doctors, nurses and enrolled nurses) rated their workload during the previous hour on a questionnaire.

Table 1 Characteristics of included study sites

All ED patient flow data were collected after the assessment period from the electronic health records (EHR) at each hospital; TakeCare™ at Karolinska University Hospital Solna and Huddinge, and Cambio Cosmic™ at Linköping University Hospital and Motala Community Hospital. Workload assessments were transcribed from paper questionnaires to digital spreadsheets.

International models

To compare the mSEAL model with established measures of crowding we computed the Occupancy rate [12] for each time point at all EDs. We also calculated the simplified ICMED (sICMED) [13] and NEDOCS [5], with some minor modifications (see below) for each time point at the two EDs (Linköping and Motala) where the needed data were available. The variable occupied ventilator/trauma bay was excluded from the present NEDOCS calculation since the data could not be reliably calculated from the EHRs. We used the short version of ICMED (sICMED) score since the variable Left without being seen was not registered systematically in the EHRs at the study sites [13]. We also excluded ambulance offload time since this does not occur in any of the study sites and very rarely in any Swedish ED. Based on the result of our SEAL derivation study [9], and recent evidence suggesting that negative effects on patient care occur mostly at high crowding levels [14], a mSEAL score in the top quartile (over 4.5) was defined as crowding.

Ethics

The study was approved by the regional ethics committee at the county of Östergötland and Stockholm.

Statistical analysis

Linear regression was used to assess correlation and stepwise backward elimination was used to derive the mSEAL model. Correlations were reported as the coefficient of determination (r2) with general qualitative descriptions [15]. Area under the receiver operating curve (AuROC) with Youden Index [16] was used to describe the models test characteristics. A two-tailed p-value of less than 0.05 was considered statistically significant. Data was imported into Pandas (v 0.23) [17] and analyzed with Python using the Scipy library (v 1.17) [18] and Statsmodels library (v. 0.10) [19].

Results

Derivation and internal validation

The mSEAL included the variables Time to MD and Patient Hours; mSEAL = 1.49 + 9.72 * Patient Hours + 0.18 * Time to MD with a range from 1 to 6. The mSEAL model showed similar correlation with the assessments (r2 = 0.47) as the original SEAL model in the derivation dataset (r2 0.47 vs 0.51) and the internal validation dataset (r2 0.69 and 0.66 vs 0.64 and 0.64).

External validation

A total of 2794 Workload assessments were collected at the four study sites on 333 (95%, 17 missing) timepoints. As can be seen in Table 2, the mSEAL score and Occupancy rate was possible to calculate at all EDs, and both showed a strong correlation to staff workload. NEDOCS compared well with both Occupancy rate and mSEAL scores, while sICMED performed worse at both hospitals. Scatterplots of scores against workload assessments were similar for all models and EDs except for sICMED that uses discrete values (Fig. 1).

Table 2 Correlations (r2) between assessed workload and crowding score at each ED
Fig. 1
figure1

Correlations between assessed workload and crowding score at each study site. mSEAL, modified Skåne Emergency Department Assessment of Patient Load. SEAL, Skåne Emergency Department Assessment of Patient Load. sICMED, short International Crowding Measure in Emergency Departments NEDOCS, National Emergency Department Overcrowding Score. * Ambulances waiting to offload patients not calculated. ** Respirator/Trauma room variable not calculated

The AuROC was calculated based on 287 of 333 (86%) available time points for mSEAL and Occupancy rate and 160 of 202 (79%) time points for sICMED and NEDOCS with values between 0.82 and 0.97. Model results with suggested cutoffs and test characteristics that predict crowding are shown in Table 3.

Table 3 AuROC and optimal model cut-off to predict an assessed workload of 4.5

Discussion

In this study, we derived and internally and externally validated the mSEAL score. Our findings indicated that mSEAL can be used to estimate ED crowding based on staff-perceived workload. The mSEAL score performed equal to the Occupancy rate and NEDOCS, and slightly better than sICMED based on their respective correlations and AuROC. Unfortunately, NEDOCS and sICMED could not be measured at two EDs due to limitations in the EHR data. With mSEAL we have the possibility to systematically measure crowding and evaluate efforts to reduce its effect on patient care. Additionally, the mSEAL model includes variables reported to the Swedish national ED registry and the National Board of Health and Welfare and has the potential of measuring crowding nationally, and also retrospectively [20, 21].

We derived the mSEAL score with the same process and data as the original SEAL model, but with the inclusion of only variables that could be retrieved at all hospitals. The mSEAL score thus includes two variables instead of four in the original SEAL score, and the additive value of mSEAL over a single measure such as Occupancy rate is probably small. However, the coefficient of determination scores were consistently better for mSEAL than for Occupancy rate at all hospitals. In situations where automatic measuring is not possible, Occupancy Rate may be a good alternative to mSEAL score at Swedish EDs.

Our results are similar to previous studies on both NEDOCS and ICMED. In a comparison of NEDOCS and the Emergency Department Work Index (EDWIN) at a single center in the US, Weiss et al. found an AuROC of 0.83 for NEDOCS [22]. When defining crowding as ambulance diversion, NEDOCS had an AuROC of 0.88 at four US hospitals [23]. The ICMED and NEDOCS models were validated against perceptions of crowding and danger at four UK hospitals with r-values of 0.73 (r2 = 0.53) and 0.77 (r2 = 0.59) respectively [10]. A study on sICMED at seven EDs in five countries showed moderate correlation with staff perception of crowding (r = 0.41, r2 = 0.17) and safety (r = 0.46, r2 = 0.21), with considerable variation between EDs [13].

There is no national consensus regarding when an ED is crowded in Sweden. To calculate the test characteristics of the mSEAL score we compared it against a dichotomized score of higher than 4.5 (crowded) or lower (non-crowded). The suggested model cutoffs in Table 3 is based on the Youden index which is the value with highest sum of sensitivity and specificity. Both cutoffs for NEDOCS (90) and OR (1.0) are close to the established values of 100 and 1.0 respectively, suggesting generalisability of an mSEAL score of 4.5 as an indication of crowding. Ideally, this cutoff should correspond to a level of crowding where patient care is compromised and morbidity and mortality increases and should be the focus of further research.

Limitations

All model scores were calculated on data from EHRs that rely on manual input from staff of time-based metrics like Time to physician. There is thus a risk of input errors or delays that may be more pronounced during crowded periods.

Data was collected in the calendar months of September–November. Although some data suggest seasonal effects on ED boarding [24], others found no seasonal effects on ED visits [25, 26]. Since any seasonal effects would probably impact all scores similarly, we find it unlikely that they affected our results.

Staff caring for high acuity patients at the time point rated their workload as soon as possible afterwards, and if this was not possible within 1 h, the ratings were not recorded. This may decrease the sensitivity to crowding of the mSEAL model. We did not systematically track the number of unrated data due to high workload but it was a rare phenomenon in the three EDs where a researcher was present at a majority of the time points.

Conclusions

The mSEAL model can measure crowding based workload in Swedish EDs with good discriminatory capacity and may be a tool to systematically evaluate crowding and help policymakers and researchers target its causes and effects. Occupancy Rate and NEDOCS are possible to measure in Swedish EDs and may be good alternatives to mSEAL.

Availability of data and materials

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Change history

Abbreviations

AuROC:

Area under the receiver operating curve

ED:

Emergency Department

EDWIN:

Emergency Department Work Index

EHR:

Electronic Health Record

ICMED:

International Crowding Metric in Emergency Department

NEDOCS:

National Emergency Department Overcrowding Score

MD:

Medical Doctor

mSEAL:

modified Swedish Emergency Department Assessment of Patient Load

SEAL:

Swedish Emergency Department Assessment of Patient Load

sICMED:

Simplified International Crowding Measure in Emergency Department

US:

United States

References

  1. 1.

    Stang AS, Crotts J, Johnson DW, Hartling L, Guttmann A. Crowding Measures Associated With the Quality of Emergency Department Care: A Systematic Review. Acad Emerg Med. 2015;22:643–56. https://doi.org/10.1111/acem.12682.

    Article  Google Scholar 

  2. 2.

    Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang L-J, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013; 61:605–611.e6. doi:10.1016/j.annemergmed.2012.10.026.

  3. 3.

    Hwang U, McCarthy ML, Aronsky D, Asplin B, Crane PW, Craven CK, et al. Measures of crowding in the emergency department: a systematic review. Acad Emerg Med. 2011; 18:527–538. doi:10.1111/j.1553-2712.2011.01054.x.

  4. 4.

    Boyle A, Coleman J, Sultan Y, Dhakshinamoorthy V, O’Keeffe J, Raut P, et al. Initial validation of the International Crowding Measure in Emergency Departments (ICMED) to measure emergency department crowding. Emerg Med J. 2015; 32: 105–8. doi:10.1136/emermed-2013-202849.

  5. 5.

    Weiss SJ, Derlet R, Arndahl J, Ernst AA, Richards J, Fernández-Frackelton M, et al. Estimating the degree of emergency department overcrowding in academic medical centers: results of the National ED Overcrowding Study (NEDOCS). Acad Emerg Med. 2004;11:38–50. https://doi.org/10.1197/S1069-6563(03)00583-9.

    Article  Google Scholar 

  6. 6.

    Pines JM, Hilton JA, Weber EJ, Alkemade AJ, Al Shabanah H, Anderson PD, et al. International Perspectives on Emergency Department Crowding: International Perspectives on ED Crowding. Acad Emerg Med. 2011;18:1358–70. https://doi.org/10.1111/j.1553-2712.2011.01235.x.

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Af Ugglas B, Djärv T, Ljungman PLS, Holzmann MJ. Association Between Hospital Bed Occupancy and Outcomes in Emergency Care: A Cohort Study in Stockholm Region, Sweden, 2012 to 2016. Ann Emerg Med. 2020;76:179–90. https://doi.org/10.1016/j.annemergmed.2019.11.009.

    Article  Google Scholar 

  8. 8.

    Wretborn J, Henricson J, Ekelund U, Wilhelms DB. Prevalence of crowding, boarding and staffing levels in Swedish emergency departments - a National Cross Sectional Study. BMC Emerg Med. 2020;20:50. https://doi.org/10.1186/s12873-020-00342-x.

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Wretborn J, Khoshnood A, Wieloch M, Ekelund U. Skåne Emergency Department Assessment of Patient Load (SEAL)-A Model to Estimate Crowding Based on Workload in Swedish Emergency Departments. PLoS One. 2015;10:e0130020. https://doi.org/10.1371/journal.pone.0130020.

    Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Boyle A, Abel G, Raut P, Austin R, Dhakshinamoorthy V, Ayyamuthu R, et al. Comparison of the International Crowding Measure in Emergency Departments (ICMED) and the National Emergency Department Overcrowding Score (NEDOCS) to measure emergency department crowding: pilot study. Emerg Med J. 2016; 33: 307–312. doi:10.1136/emermed-2014-203616.

  11. 11.

    Wang H, Ojha RP, Robinson RD, Jackson BE, Shaikh SA, Cowden CD, et al. Optimal Measurement Interval for Emergency Department Crowding Estimation Tools. Ann Emerg Med. 2017; 70:632–639.e4. doi:10.1016/j.annemergmed.2017.04.012.

  12. 12.

    McCarthy ML, Aronsky D, Jones ID, Miner JR, Band RA, Baren JM, et al. The Emergency Department Occupancy Rate: A Simple Measure of Emergency Department Crowding? Ann Emerg Med. 2008; 51:15–24.e2. doi:10.1016/j.annemergmed.2007.09.003.

  13. 13.

    Boyle A, Atkinson P, Basaure Verdejo C, Chan E, Clouston R, Gilligan P, et al. Validation of the short form of the International Crowding Measure in Emergency Departments: an international study. Eur J Emerg Med. 2018. doi: 10.1097/MEJ.0000000000000579.

  14. 14.

    af Ugglas B, Djärv T, Ljungman PLS, Holzmann MJ. Emergency department crowding associated with increased 30-day mortality: a cohort study in Stockholm Region, Sweden, 2012 to 2016. J Am Coll Emerg Phys Open. 2020. doi:10.1002/emp2.12243.

  15. 15.

    Taylor R. Interpretation of the Correlation Coefficient: A Basic Review. J Diagn Med Sonogr. 1990;6:35–9. https://doi.org/10.1177/875647939000600106.

    Article  Google Scholar 

  16. 16.

    Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32–35. doi: 3.0.CO;2-3">10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3.

  17. 17.

    McKinney W. Data structures for statistical computing in python. In: Proceedings of the 9th Python in Science Conference. 2010. p. 51–6. https://doi.org/10.25080/Majora-92bf1922-00a.

  18. 18.

    Millman KJ, Aivazis M. Python for Scientists and Engineers. Comput Sci Eng. 2011;13:9–12. https://doi.org/10.1109/MCSE.2011.36.

    Article  Google Scholar 

  19. 19.

    Seabold S, Perktold J. Statsmodels: Econometric and statistical modeling with python. In: Proceedings of the 9th Python in Science Conference. 2010. https://doi.org/10.25080/Majora-92bf1922-011.

  20. 20.

    Ekelund U, Kurland L, Eklund F, Torkki P, Letterstål A, Lindmarker P, et al. Patient throughput times and inflow patterns in Swedish emergency departments. A basis for ANSWER, A National SWedish Emergency Registry. Scand J Trauma Resusc Emerg Med. 2011;19:37. https://doi.org/10.1186/1757-7241-19-37.

  21. 21.

    Ruge T, Granberg M, Helmers SB. Väntetider och patientflöden på akutmottagningar – Rapport februari 2017. 2017. https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/artikelkatalog/ovrigt/2017-2-16.pdf. Accessed 22 Mar 2019.

  22. 22.

    Weiss SJ, Ernst AA, Nick TG. Comparison of the National Emergency Department Overcrowding Scale and the Emergency Department Work Index for quantifying emergency department crowding. Acad Emerg Med. 2006;13:513–8. https://doi.org/10.1197/j.aem.2005.12.009.

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23.

    Hoot NR, Zhou C, Jones I, Aronsky D. Measuring and forecasting emergency department crowding in real time. Ann Emerg Med. 2007;49:747–55. https://doi.org/10.1016/j.annemergmed.2007.01.017.

    Article  PubMed  PubMed Central  Google Scholar 

  24. 24.

    Andrew Taylor R, Venkatesh A, Parwani V, Chekijian S, Shapiro M, Oh A, et al. Applying advanced analytics to guide emergency department operational decisions: A proof-of-concept study examining the effects of boarding. Am J Emerg Med. 2018;36:1534–9. https://doi.org/10.1016/j.ajem.2018.01.011.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  25. 25.

    Batal H, Tench J, McMillan S, Adams J, Mehler PS. Predicting patient visits to an urgent care clinic using calendar variables. Acad Emerg Med. 2001;8:48–53. https://doi.org/10.1111/j.1553-2712.2001.tb00550.x.

    CAS  Article  Google Scholar 

  26. 26.

    Marcilio I, Hajat S, Gouveia N. Forecasting daily emergency department visits using calendar variables and ambient temperature readings. Acad Emerg Med. 2013;20:769–77. https://doi.org/10.1111/acem.12182.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors thank the clinical staff at the study sites for their assistance in the data acquisition.

Funding

This work was supported by two grants from Region Östergötland to author DBW (LIO-532001 and LIO-700271), and from Region Skåne to author UE. The funding bodies had no role or influence over any aspect of this study. Open Access funding provided by Lund University.

Author information

Affiliations

Authors

Contributions

JW, DW, TR and UE conceived the study and obtained the ethical permit. JW, DBW and UE designed the trial. HS and JW coordinated data collection and managed the data, with the assistance of DBW, TR and UE. TR, DBW and UE obtained the research funding. DBW and TR supervised the conduct of the trial. JW drafted the manuscript. All authors contributed substantially to its revision. JW takes responsibility for the paper as a whole.

Corresponding author

Correspondence to Jens Wretborn.

Ethics declarations

Ethics approval and consent to participate

The part of the study pertaining to the Linköping University Hospital and Motala Community Hospital was approved by the regional ethics committee at the county of Östergötland. The data from the Karolinska Hospital in Huddinge and Solna was part of a quality improvement initiative only including anonymised data, and as such is exempt from ethical approval in Sweden.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wretborn, J., Starkenberg, H., Ruge, T. et al. Validation of the modified Skåne emergency department assessment of patient load (mSEAL) model for emergency department crowding and comparison with international models; an observational study. BMC Emerg Med 21, 21 (2021). https://doi.org/10.1186/s12873-021-00414-6

Download citation

Keywords

  • Emergency department
  • Crowding
  • Boarding
\