- Research article
- Open Access
Validation of the modified Skåne emergency department assessment of patient load (mSEAL) model for emergency department crowding and comparison with international models; an observational study
BMC Emergency Medicine volume 21, Article number: 21 (2021)
Emergency Department crowding is associated with increased morbidity and mortality but no measure of crowding has been validated in Sweden. We have previously derived and internally validated the Skåne Emergency Department Assessment of Patient Load (SEAL) score as a measure of crowding in Emergency Departments (ED) in a large regional healthcare system in Sweden. Due to differences in electronic health records (EHRs) between health care systems in Sweden, all variables in the original SEAL-score could not be measured reliably nationally. We aimed to derive and validate a modified SEAL (mSEAL) model and to compare it with established international measures of crowding.
This was an observational cross sectional study at four EDs in Sweden. All clinical staff assessed their workload (1–6 where 6 is the highest workload) at 5 timepoints each day. We used linear regression with stepwise backward elimination on the original SEAL dataset to derive and internally validate the mSEAL score against staff workload assessments. We externally validated the mSEAL at four hospitals and compared it with the National Emergency Department Overcrowding Score (NEDOCS), the simplified International Crowding Measure in Emergency Department (sICMED), and Occupancy Rate. Area under the receiver operating curve (AuROC) and coefficient of determination was used to compare crowding models. Crowding was defined as an average workload of 4.5 or higher.
The mSEAL score contains the variables Patient Hours and Time to physician and showed strong correlation with crowding in the derivation (r2 = 0.47), internal validation (r2 = 0.64 and 0.69) and in the external validation (r2 = 0.48 to 0.60). AuROC scores for crowding in the external validation were 0.91, 0.90, 0.97 and 0.80 for mSEAL, Occupancy Rate, NEDOCS and sICMED respectively.
The mSEAL model can measure crowding based on workload in Swedish EDs with good discriminatory capacity and has the potential to systematically evaluate crowding and help policymakers and researchers target its causes and effects. In Swedish EDs, Occupancy Rate and NEDOCS are good alternatives to measure crowding based on workload.
Emergency Department (ED) crowding is a prevalent problem in many health care systems and associated with increased morbidity, mortality and decreased quality of care [1, 2]. There are several measures of crowding, from simple numeric scores to more complex models which incorporate different measures into a single score . The rationale is that complex models encompass different aspects of crowding and are thought to provide better information compared to single variables such as Occupancy Rate [4, 5]. Crowding in Swedish EDs has previously been limited , but has become a problem in recent years, and several research projects on the topic have been initiated [7, 8]. However there is no definition or validated measure of crowding in Sweden which limits the efforts to detect and systematically prevent its negative effects on patient care. Internationally the National Emergency Department Overcrowding Score (NEDOCS) have been derived and validated in the US using staff perception of crowding and generates a score from 1 to 200, with overcrowding defined as a score over 100 . The International Crowding Measure in Emergency Department (ICMED) was derived in the UK against staff perception of crowding and danger. Points are assigned to different aspects of ED care on a scale from 1 to 7, and a score of 3 or higher has a high predictive ability for crowding . Both scores include variables that are not readily available in the ED and there are concerns whether they could be applied to the Swedish healthcare system .
Since there are no validated measures of crowding in Sweden, we derived and validated the Skåne Emergency Department Assessment of Patient Load (SEAL) score in a large regional health care system in southern Sweden . We used staff perception of workload as a measure of ED crowding since it is reasonable to assume that a substantial part of the negative effects of crowding are due to high staff workload, and since similar measures have been shown to be reliable in other crowding models like NEDOCS and ICMED [4, 5]. The score is intended for automated calculation based on data from the EHR and internal validation has shown consistent results . So far, the SEAL score has not been externally validated, and its utility has not been compared with existing international crowding models.
Sweden, with a population of 10 million, has a universal publicly funded health care system granting emergency care with a small co-payment at 72 EDs spanning from small rural EDs to large urban academic EDs. The health care is organized into 21 health care systems with different EHRs and ED support systems. Because of differences in the EHRs, all variables in the original SEAL score cannot be measured reliably in many Swedish health care systems at this time. For optimal usability and generalisability, a crowding model needs to include measures readily available in the EHR at many EDs.
In the present study, we aimed to derive and externally validate a modified SEAL model (mSEAL) to measure ED crowding based on variables reliably available across different EHRs, and to compare it with established international measures of crowding.
Derivation and validation of mSEAL
We used linear regression analysis with stepwise backward elimination on the original SEAL-dataset . The SEAL model includes the variables Patient Hours, High acuity Patients, Time waiting for a physician and Occupancy rate. In short, Patient hours is the sum of time all patients spent in the ED during the past hour, measured in hours, and has the largest impact on the model score. High Acuity Patients is the proportion of high (highest and second highest out of five categories) acuity patients at the ED. Time waiting for a physician is the time from triage and nursing procedures to first physician contact. Occupancy rate is the number of ED patients divided by the number of ED treatment beds.
Since High Acuity Patients and Time waiting for a physician could not be reliably collected at all EDs, these were left out and exchanged for the new variable time to physician (measuring the time from registration in the ED to first physician contact) to form the mSEAL score. The mSEAL was then internally validated in a separate data collection from two of the five hospitals in the original study, as well as externally validated at four EDs, as described below.
We collected data at four EDs in Sweden (Table 1) during September to November 2017; Three EDs at academic teaching hospitals and one at a rural community hospital in two different healthcare systems. Workload assessments, rated on a Likert scale from 1 (no workload) to 6 (very high workload), were collected from each included ED at 5 pre-defined time points per day (08:00, 12:00, 16:00, 20:00, 23:00). These time points were selected to represent known periods of both high and low levels of crowding, and were separated in time to reduce the risk of temporal correlation between assessment and scores [10, 11]. All working staff (doctors, nurses and enrolled nurses) rated their workload during the previous hour on a questionnaire.
All ED patient flow data were collected after the assessment period from the electronic health records (EHR) at each hospital; TakeCare™ at Karolinska University Hospital Solna and Huddinge, and Cambio Cosmic™ at Linköping University Hospital and Motala Community Hospital. Workload assessments were transcribed from paper questionnaires to digital spreadsheets.
To compare the mSEAL model with established measures of crowding we computed the Occupancy rate  for each time point at all EDs. We also calculated the simplified ICMED (sICMED)  and NEDOCS , with some minor modifications (see below) for each time point at the two EDs (Linköping and Motala) where the needed data were available. The variable occupied ventilator/trauma bay was excluded from the present NEDOCS calculation since the data could not be reliably calculated from the EHRs. We used the short version of ICMED (sICMED) score since the variable Left without being seen was not registered systematically in the EHRs at the study sites . We also excluded ambulance offload time since this does not occur in any of the study sites and very rarely in any Swedish ED. Based on the result of our SEAL derivation study , and recent evidence suggesting that negative effects on patient care occur mostly at high crowding levels , a mSEAL score in the top quartile (over 4.5) was defined as crowding.
The study was approved by the regional ethics committee at the county of Östergötland and Stockholm.
Linear regression was used to assess correlation and stepwise backward elimination was used to derive the mSEAL model. Correlations were reported as the coefficient of determination (r2) with general qualitative descriptions . Area under the receiver operating curve (AuROC) with Youden Index  was used to describe the models test characteristics. A two-tailed p-value of less than 0.05 was considered statistically significant. Data was imported into Pandas (v 0.23)  and analyzed with Python using the Scipy library (v 1.17)  and Statsmodels library (v. 0.10) .
Derivation and internal validation
The mSEAL included the variables Time to MD and Patient Hours; mSEAL = 1.49 + 9.72 * Patient Hours + 0.18 * Time to MD with a range from 1 to 6. The mSEAL model showed similar correlation with the assessments (r2 = 0.47) as the original SEAL model in the derivation dataset (r2 0.47 vs 0.51) and the internal validation dataset (r2 0.69 and 0.66 vs 0.64 and 0.64).
A total of 2794 Workload assessments were collected at the four study sites on 333 (95%, 17 missing) timepoints. As can be seen in Table 2, the mSEAL score and Occupancy rate was possible to calculate at all EDs, and both showed a strong correlation to staff workload. NEDOCS compared well with both Occupancy rate and mSEAL scores, while sICMED performed worse at both hospitals. Scatterplots of scores against workload assessments were similar for all models and EDs except for sICMED that uses discrete values (Fig. 1).
The AuROC was calculated based on 287 of 333 (86%) available time points for mSEAL and Occupancy rate and 160 of 202 (79%) time points for sICMED and NEDOCS with values between 0.82 and 0.97. Model results with suggested cutoffs and test characteristics that predict crowding are shown in Table 3.
In this study, we derived and internally and externally validated the mSEAL score. Our findings indicated that mSEAL can be used to estimate ED crowding based on staff-perceived workload. The mSEAL score performed equal to the Occupancy rate and NEDOCS, and slightly better than sICMED based on their respective correlations and AuROC. Unfortunately, NEDOCS and sICMED could not be measured at two EDs due to limitations in the EHR data. With mSEAL we have the possibility to systematically measure crowding and evaluate efforts to reduce its effect on patient care. Additionally, the mSEAL model includes variables reported to the Swedish national ED registry and the National Board of Health and Welfare and has the potential of measuring crowding nationally, and also retrospectively [20, 21].
We derived the mSEAL score with the same process and data as the original SEAL model, but with the inclusion of only variables that could be retrieved at all hospitals. The mSEAL score thus includes two variables instead of four in the original SEAL score, and the additive value of mSEAL over a single measure such as Occupancy rate is probably small. However, the coefficient of determination scores were consistently better for mSEAL than for Occupancy rate at all hospitals. In situations where automatic measuring is not possible, Occupancy Rate may be a good alternative to mSEAL score at Swedish EDs.
Our results are similar to previous studies on both NEDOCS and ICMED. In a comparison of NEDOCS and the Emergency Department Work Index (EDWIN) at a single center in the US, Weiss et al. found an AuROC of 0.83 for NEDOCS . When defining crowding as ambulance diversion, NEDOCS had an AuROC of 0.88 at four US hospitals . The ICMED and NEDOCS models were validated against perceptions of crowding and danger at four UK hospitals with r-values of 0.73 (r2 = 0.53) and 0.77 (r2 = 0.59) respectively . A study on sICMED at seven EDs in five countries showed moderate correlation with staff perception of crowding (r = 0.41, r2 = 0.17) and safety (r = 0.46, r2 = 0.21), with considerable variation between EDs .
There is no national consensus regarding when an ED is crowded in Sweden. To calculate the test characteristics of the mSEAL score we compared it against a dichotomized score of higher than 4.5 (crowded) or lower (non-crowded). The suggested model cutoffs in Table 3 is based on the Youden index which is the value with highest sum of sensitivity and specificity. Both cutoffs for NEDOCS (90) and OR (1.0) are close to the established values of 100 and 1.0 respectively, suggesting generalisability of an mSEAL score of 4.5 as an indication of crowding. Ideally, this cutoff should correspond to a level of crowding where patient care is compromised and morbidity and mortality increases and should be the focus of further research.
All model scores were calculated on data from EHRs that rely on manual input from staff of time-based metrics like Time to physician. There is thus a risk of input errors or delays that may be more pronounced during crowded periods.
Data was collected in the calendar months of September–November. Although some data suggest seasonal effects on ED boarding , others found no seasonal effects on ED visits [25, 26]. Since any seasonal effects would probably impact all scores similarly, we find it unlikely that they affected our results.
Staff caring for high acuity patients at the time point rated their workload as soon as possible afterwards, and if this was not possible within 1 h, the ratings were not recorded. This may decrease the sensitivity to crowding of the mSEAL model. We did not systematically track the number of unrated data due to high workload but it was a rare phenomenon in the three EDs where a researcher was present at a majority of the time points.
The mSEAL model can measure crowding based workload in Swedish EDs with good discriminatory capacity and may be a tool to systematically evaluate crowding and help policymakers and researchers target its causes and effects. Occupancy Rate and NEDOCS are possible to measure in Swedish EDs and may be good alternatives to mSEAL.
Availability of data and materials
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Area under the receiver operating curve
Emergency Department Work Index
Electronic Health Record
International Crowding Metric in Emergency Department
National Emergency Department Overcrowding Score
modified Swedish Emergency Department Assessment of Patient Load
Swedish Emergency Department Assessment of Patient Load
Simplified International Crowding Measure in Emergency Department
Stang AS, Crotts J, Johnson DW, Hartling L, Guttmann A. Crowding Measures Associated With the Quality of Emergency Department Care: A Systematic Review. Acad Emerg Med. 2015;22:643–56. https://doi.org/10.1111/acem.12682.
Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang L-J, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013; 61:605–611.e6. doi:10.1016/j.annemergmed.2012.10.026.
Hwang U, McCarthy ML, Aronsky D, Asplin B, Crane PW, Craven CK, et al. Measures of crowding in the emergency department: a systematic review. Acad Emerg Med. 2011; 18:527–538. doi:10.1111/j.1553-2712.2011.01054.x.
Boyle A, Coleman J, Sultan Y, Dhakshinamoorthy V, O’Keeffe J, Raut P, et al. Initial validation of the International Crowding Measure in Emergency Departments (ICMED) to measure emergency department crowding. Emerg Med J. 2015; 32: 105–8. doi:10.1136/emermed-2013-202849.
Weiss SJ, Derlet R, Arndahl J, Ernst AA, Richards J, Fernández-Frackelton M, et al. Estimating the degree of emergency department overcrowding in academic medical centers: results of the National ED Overcrowding Study (NEDOCS). Acad Emerg Med. 2004;11:38–50. https://doi.org/10.1197/S1069-6563(03)00583-9.
Pines JM, Hilton JA, Weber EJ, Alkemade AJ, Al Shabanah H, Anderson PD, et al. International Perspectives on Emergency Department Crowding: International Perspectives on ED Crowding. Acad Emerg Med. 2011;18:1358–70. https://doi.org/10.1111/j.1553-2712.2011.01235.x.
Af Ugglas B, Djärv T, Ljungman PLS, Holzmann MJ. Association Between Hospital Bed Occupancy and Outcomes in Emergency Care: A Cohort Study in Stockholm Region, Sweden, 2012 to 2016. Ann Emerg Med. 2020;76:179–90. https://doi.org/10.1016/j.annemergmed.2019.11.009.
Wretborn J, Henricson J, Ekelund U, Wilhelms DB. Prevalence of crowding, boarding and staffing levels in Swedish emergency departments - a National Cross Sectional Study. BMC Emerg Med. 2020;20:50. https://doi.org/10.1186/s12873-020-00342-x.
Wretborn J, Khoshnood A, Wieloch M, Ekelund U. Skåne Emergency Department Assessment of Patient Load (SEAL)-A Model to Estimate Crowding Based on Workload in Swedish Emergency Departments. PLoS One. 2015;10:e0130020. https://doi.org/10.1371/journal.pone.0130020.
Boyle A, Abel G, Raut P, Austin R, Dhakshinamoorthy V, Ayyamuthu R, et al. Comparison of the International Crowding Measure in Emergency Departments (ICMED) and the National Emergency Department Overcrowding Score (NEDOCS) to measure emergency department crowding: pilot study. Emerg Med J. 2016; 33: 307–312. doi:10.1136/emermed-2014-203616.
Wang H, Ojha RP, Robinson RD, Jackson BE, Shaikh SA, Cowden CD, et al. Optimal Measurement Interval for Emergency Department Crowding Estimation Tools. Ann Emerg Med. 2017; 70:632–639.e4. doi:10.1016/j.annemergmed.2017.04.012.
McCarthy ML, Aronsky D, Jones ID, Miner JR, Band RA, Baren JM, et al. The Emergency Department Occupancy Rate: A Simple Measure of Emergency Department Crowding? Ann Emerg Med. 2008; 51:15–24.e2. doi:10.1016/j.annemergmed.2007.09.003.
Boyle A, Atkinson P, Basaure Verdejo C, Chan E, Clouston R, Gilligan P, et al. Validation of the short form of the International Crowding Measure in Emergency Departments: an international study. Eur J Emerg Med. 2018. doi: 10.1097/MEJ.0000000000000579.
af Ugglas B, Djärv T, Ljungman PLS, Holzmann MJ. Emergency department crowding associated with increased 30-day mortality: a cohort study in Stockholm Region, Sweden, 2012 to 2016. J Am Coll Emerg Phys Open. 2020. doi:10.1002/emp2.12243.
Taylor R. Interpretation of the Correlation Coefficient: A Basic Review. J Diagn Med Sonogr. 1990;6:35–9. https://doi.org/10.1177/875647939000600106.
Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32–35. doi: 3.0.CO;2-3">10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3.
McKinney W. Data structures for statistical computing in python. In: Proceedings of the 9th Python in Science Conference. 2010. p. 51–6. https://doi.org/10.25080/Majora-92bf1922-00a.
Millman KJ, Aivazis M. Python for Scientists and Engineers. Comput Sci Eng. 2011;13:9–12. https://doi.org/10.1109/MCSE.2011.36.
Seabold S, Perktold J. Statsmodels: Econometric and statistical modeling with python. In: Proceedings of the 9th Python in Science Conference. 2010. https://doi.org/10.25080/Majora-92bf1922-011.
Ekelund U, Kurland L, Eklund F, Torkki P, Letterstål A, Lindmarker P, et al. Patient throughput times and inflow patterns in Swedish emergency departments. A basis for ANSWER, A National SWedish Emergency Registry. Scand J Trauma Resusc Emerg Med. 2011;19:37. https://doi.org/10.1186/1757-7241-19-37.
Ruge T, Granberg M, Helmers SB. Väntetider och patientflöden på akutmottagningar – Rapport februari 2017. 2017. https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/artikelkatalog/ovrigt/2017-2-16.pdf. Accessed 22 Mar 2019.
Weiss SJ, Ernst AA, Nick TG. Comparison of the National Emergency Department Overcrowding Scale and the Emergency Department Work Index for quantifying emergency department crowding. Acad Emerg Med. 2006;13:513–8. https://doi.org/10.1197/j.aem.2005.12.009.
Hoot NR, Zhou C, Jones I, Aronsky D. Measuring and forecasting emergency department crowding in real time. Ann Emerg Med. 2007;49:747–55. https://doi.org/10.1016/j.annemergmed.2007.01.017.
Andrew Taylor R, Venkatesh A, Parwani V, Chekijian S, Shapiro M, Oh A, et al. Applying advanced analytics to guide emergency department operational decisions: A proof-of-concept study examining the effects of boarding. Am J Emerg Med. 2018;36:1534–9. https://doi.org/10.1016/j.ajem.2018.01.011.
Batal H, Tench J, McMillan S, Adams J, Mehler PS. Predicting patient visits to an urgent care clinic using calendar variables. Acad Emerg Med. 2001;8:48–53. https://doi.org/10.1111/j.1553-2712.2001.tb00550.x.
Marcilio I, Hajat S, Gouveia N. Forecasting daily emergency department visits using calendar variables and ambient temperature readings. Acad Emerg Med. 2013;20:769–77. https://doi.org/10.1111/acem.12182.
The authors thank the clinical staff at the study sites for their assistance in the data acquisition.
This work was supported by two grants from Region Östergötland to author DBW (LIO-532001 and LIO-700271), and from Region Skåne to author UE. The funding bodies had no role or influence over any aspect of this study. Open Access funding provided by Lund University.
Ethics approval and consent to participate
The part of the study pertaining to the Linköping University Hospital and Motala Community Hospital was approved by the regional ethics committee at the county of Östergötland. The data from the Karolinska Hospital in Huddinge and Solna was part of a quality improvement initiative only including anonymised data, and as such is exempt from ethical approval in Sweden.
Consent for publication
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Wretborn, J., Starkenberg, H., Ruge, T. et al. Validation of the modified Skåne emergency department assessment of patient load (mSEAL) model for emergency department crowding and comparison with international models; an observational study. BMC Emerg Med 21, 21 (2021). https://doi.org/10.1186/s12873-021-00414-6
- Emergency department