- Research article
- Open Access
Application of the emergency medical services trigger tool to measure adverse events in prehospital emergency care: a time series analysis
BMC Emergency Medicine volume 18, Article number: 47 (2018)
Emergency Care has previously been identified as an area of significant concern regarding the prevalence of Adverse Events (AEs). However, the majority of this focus has been on the in-hospital setting, with little understanding of the identification and incidence of AEs in the prehospital environment.
The early development and testing of Emergency Medical Services (EMS) specific triggers for the identification of AEs and Harm has been previously described. To operationalise the Emergency Medical Services Trigger Tool (EMSTT), the processes developed by the Institute for Healthcare Improvement for use with the Global Trigger Tool were adapted to a prehospital emergency care setting. These were then applied using a stepwise approach to the analysis of 36 consecutive samples of patient care records over an 18-month period (n = 710). Inter-rater reliability was measured for each trigger item and level of Harm classification. Total Triggers per 10,000 Patient Encounters, AEs per 10,000 Patient Encounters and Harm per 10,000 Patient Encounters were measured. All measures were plotted on Statistical Process Control Charts.
There was a high level of inter-rater agreement across all items (range: 85.6–100%). The EMSTT found an average rate of 8.20 Triggers per 10,000 Patient Encounters, 2.48 AEs per 10,000 Patient Encounters and 0.34 Harm events per 10,000 Patient Encounters. Three triggers: Change in Systolic Blood Pressure Greater Than 20%; Temp > 38 °C without subsequent reduction; and SpO2 < 94% without supplemental Oxygen or SpO2 < 85% without assisted ventilation accounted for 93% (n = 180) of the triggers found throughout the longitudinal analysis.
With sufficient focus on implementation and data collection, as well as the inclusion of a contextually relevant system for classifying AE/Harm, the EMSTT represents a potentially successful strategy towards identifying the rate of AEs within EMS across a large patient population with limited commitment of time and resources.
The identification of adverse events (AE), defined by the Institute of Medicine as “an injury caused by medical management rather than the underlying condition of the patient”  has become an essential component of patient safety and clinical governance within healthcare [1, 2]. Emergency care has previously been identified as an area of significant concern regarding the prevalence of AEs . However, the majority of this focus has been on the in-hospital setting, with little understanding of the identification and incidence of AEs in the prehospital environment. The potential for AEs to occur within emergency medical services (EMS) is significant, given that prehospital clinicians are required to make prompt decisions, with little to no supporting patient information, and implement care with limited resources, in a multitude of challenging environments.
A number of methods that have been employed to monitor for AEs in healthcare rely on either voluntary reporting systems, direct observation, complaints, mortality and morbidity review, or patient care documentation review [3,4,5,6]. While these methods may be effective for specific patients or defined high risk procedures, there is little evidence to suggest they provide a comprehensive or robust system of AE detection [3,4,5,6]. A novel approach known as the Trigger Tool (TT) methodology has been shown to be a more time-effective, cost-effective and sensitive means of identifying AEs when compared with more traditional methods [7,8,9].
The early development and testing of EMS specific triggers for the identification of AEs and Harm has been previously described . The primary focus for these triggers was on the the low risk/high frequency case cohort of an EMS organisation. High risk/low frequency cases such as those that make up a key clinical care pathway (e.g. ST-elevation myocardial infarction); high-risk procedures (e.g., surgical airway); the administration of high-risk medication (e.g., paralytic agents); infrequent procedures (e.g., intraosseous cannulation) and high acuity inter-facility transport cases were excluded given that they are generally included as part of a 100% audit framework, as a result of their high potential for AEs and harm. The remaining low risk/high frequency cases, which generally make up the significant majority of EMS calls, are often neglected from audit, and as such, were the target population. In addition, the focus of audit in high risk cases is largely aimed at identifying individual occurrences of AEs and harm for case/risk management purposes, whereas the purpose of the EMS trigger tool is to act as a global measure of system performance and to facilitate general system improvement.
The aim of this study was to test the use of EMS triggers, using a sampling and review methodology, and to report on the incidence rates and types of AEs and harm they identify.
The EMS triggers were previously developed over several derivation rounds and a single large sample analysis for performance assessment/test characteristics . As a retrospective sampling strategy for the targeted identification of cases at risk for potential AEs and harm, the EMS triggers demonstrated both a superior sensitivity and specificity compared to a random sampling strategy for audit case identification (See Table 1 for EMS Trigger Items).
In order to operationalise the EMS triggers, we adapted the processes developed by the Institute for Healthcare Improvement (IHI) for use with the Global Trigger Tool (GTT) , to an EMS setting. The adapted EMS trigger tool (EMSTT) process was applied using a stepwise approach, outlined by Adler et al. in their implementation of the GTT , to the analysis of 36 consecutive samples of patient care records over an 18-month period (n = 710).
The study was conducted within the Hamad Medical Corporation Ambulance Service (HMCAS), the government-funded national ambulance service of Qatar. HMCAS is a two-tiered provider (Ambulance Paramedic and Critical Care Paramedic) that serves a population of approximately 2.6 million people with an average daily call rate of between 600 and 700 emergency and non-emergency calls. At the time of writing, HMCAS was transitioning from a paper-based care record system to an electronic patient care record.
Institutional review board
Ethical approval to conduct the study was granted by the Medical Research Centre of the Hamad Medical Corporation, Qatar.
Trigger tool methodology
Consistent with the sampling strategy of the GTT, the EMSTT process utilises a sampling methodology of small consecutive samples over time. Cases considered for analysis included those where there was patient contact with EMS and for whom a Patient Care Record (PCR) was generated. Excluded PCRs included: those reviewed as part of a key clinical care pathway (e.g., ST-Elevation Myocardial Infarction); high-risk procedures (e.g., surgical airway); the administration of high-risk medication (e.g., paralytic agents); infrequent procedures (e.g., intraosseous cannulation); and inter-facility transport records. Of the remaining cases, two samples per month were selected by random number table for the 18-month test period (one sample for each half of the month). Each sample consisted of 30 PCRs, with the intent to review the first 20 cases, and the additional 10 PCRs included in case any of the first 20 met exclusion criteria (See Additional files 1 and 2). The sampling process was provided to a documentation clerk, who identified the PCRs meeting inclusion criteria, and supplied paper copies to the primary reviewers for review.
Each review round was conducted independently by two primary reviewers (IH, BP) during scheduled and protected review time. The primary reviewers were each operational Critical Care Paramedics employed by HMCAS, with > 15 years EMS experience each, who were additionally involved in clinical governance and education within the service. Individual record reviews were limited to 10 min each. Following each review round, the two primary reviewers met to compare findings, reach consensus, and summarize the results. In cases where consensus could not be reached, a third reviewer, the HMCAS Medical Director or Head of Professions (LAS/NC) were consulted to determine an outcome. Each review round was part of an iterative process and allowed for continued refinement of the EMSTT methodology and data collection processes (See Additional files 1 and 2).
Data collection process
Each record was manually reviewed for the presence of triggers only. If a trigger was found, the record was further reviewed for the occurrence of AEs and/or harm. Records that did not contain a trigger were not reviewed further. The National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) classification system was used to categorise AEs and harm for EMSTT positive cases. Given the in-hospital focus of the MERP system, Category C was modified to include cases where it was unclear if harm had occurred given the short duration of care provided by EMS (Table 2) . Beginning with Sample 10 (Month 5) an additional, EMS specific classification system developed by Patterson et al. , the Adverse Event Severity Rating Index, was introduced to run concurrently with the NCC MERP system (Table 2). This allowed a fit for purpose EMS-specific system to be used for AE classification, as opposed to relying solely on the in-hospital focused NCC MERP.
Consecutive sample analysis
The EMSTT was applied over an 18-month period as part of a consecutive sample analysis using the operational processes reported above. Records for review were identified via the HMCAS centralised electronic database (Microsoft Access 2010, Redwood, WA) and cases meeting exclusion criteria were removed prior to sampling. All data was captured on a standardised data capture template and summary report template (Microsoft Excel 2010, Redwood, WA) for analysis and reporting.
Data analysis and reporting process
Inter-rater reliability was measured for each trigger item and level of classification overall. In addition, univariate descriptive analysis was conducted for all continuous and categorical variables (i.e.: trigger items and levels of harm classification). Lastly, a reporting standard was developed that included outcome measures for future potential benchmarking purposes and data visualisation methods for analysis and performance monitoring. The three outcome measures identified included: Triggers per 10,000 Patient Encounters, AEs per 10,000 Patient Encounters and Harm per 10,000 Patient Encounters. For the purpose of the reporting standard, a patient encounter was defined as: “An interaction between a patient and EMS healthcare provider(s) for the purpose of providing healthcare service(s) or assessing the health status of a patient.” All outcome measures were calculated and plotted on Statistical Process Control (SPC) Charts using Minitab Version 17 (2010, State College PA) and employed the Nelson Rules for detecting special cause variation  SPC is a branch of statistics that combines time series analysis methods with graphical presentation of data to understand variation and yield insight into whether the variation is due to chance or assignable causes . The primary tool of SPC, the control chart, provides researchers and practitioners with a method of better understanding and communicating data from healthcare improvement efforts by enabling the study of variation and differentiation of special from common cause variation . The data presented in this study meets all requirements associated with the use of U-charts .
The EMSTT identified 194 individual triggers among the 720 sampled PCRs. There was a high level of inter-rater agreement across all items ranging from 85.6–100% depending on the trigger item (Table 3). Out of a total of 40 items analysed, 4 were observed to have poor reliability (κ < 0.41), 8 were observed to have moderate reliability (κ = 0.41–0.6), 3 substantial reliability (κ = 0.61–0.8), and 14 near perfect reliability (κ = 0.81–1) . Six cases in total required referral to a third reviewer, all of which were from the Clinical group. All disagreements were primarily attributed to PCR illegibility.
Consistent with the results reported in the development of the EMSTT, the Clinical group made up the majority of triggers found in our analysis [n = 190 (97.9%)] (Table 4).Three triggers: Change in Systolic Blood Pressure Greater Than 20%; Temp > 38 °C without subsequent reduction; and SpO2 < 94% without supplemental Oxygen or SpO2 < 85% without assisted ventilation accounted for 92.8% (n = 180) of the triggers found throughout the analysis. Change in Systolic Blood Pressure Greater Than 20% was the most common trigger amongst the Clinical group, as well as overall [n = 108 (55.7%)].
The three primary outcomes measures were calculated and plotted on U charts for the study period. An average rate of 8.20 Triggers per 10,000 Patient Encounters (Fig. 1), 2.48 AEs per 10,000 Patient Encounters (Fig. 2) and 0.34 Harm events per 10,000 Patient Encounters (Fig. 3) were found during the analysis. While one sample generated data outside the control limits in the Harm events per 10,000 Patient Encounters chart, no obvious attributable cause for this variation could be immediately identified.
When classifying harm using the NCC MERP Category A (Circumstances or Events that have the capacity to cause error) was the most common assigned class [n = 147 (20.7%)] (Table 5). In total, only three NCC MERP categories were assigned to all cases in which triggers were found: Category A, Category B (An error occurred but the error did not reach the patient) [n = 36 (5.1%)]; and Category C (An error occurred that reached the patient, but did not cause patient harm) [n = 11 (1.5%)], modified in this study to include cases in which it was unclear if harm had occurred due to the short duration of care provided by EMS. As with the development of the EMSTT, for reporting and analysis purposes, the latter classification was counted as positive for the occurrence of Harm.
When classifying harm using the AE Severity Rating Index, implemented with the fifth sample, Rating Code 7 (No AE identified) was the most common reported class in trigger positive cases [n = 67 (12%)] (Table 6). Similar to the NCC MERP system, three categories made up the majority of assigned categories in trigger positive cases, including: Code 7; Code 6 (AE with potential to cause harm with no fault) [n = 11 (2%)]; and Code 4 (AE with potential to cause harm as a result of commission) [n = 11 (2%)].
Our study separated the definitions of AEs and harm into separate concepts, aligning the definition of AE more synonymously with error. However, there is a recent trend towards the definition of AE to be used interchangeably and associated more with the definition of harm. The reasons for this are two-fold. Firstly, error reporting has become less prominent, given its perceived “dilution” of the reporting on harm events. In addition, early results using the trigger tool methodology reported a relatively low conversion of error into harm events, reducing the emphasis on reporting the incidence of error, and focusing more on identifying actual cases involving patient harm . Finally, harm can occur in the absence of an error in the delivery of a patient’s care. These events began to be referred to as AEs and were seen as synonymous with harm given their untoward outcome, despite the potential absence of a trigger or error in patient care .
In order to account for this recent change in the understanding of AEs, an additional measure was calculated to align with the evolving change in thinking towards these definitions - Modified AEs per 10,000 Patient Encounters values were calculated by adding AEs and harm, and the results plotted on a U chart for comparison. While this has the potential to skew the results observed for this study, the value of each of these modified measures remained relatively low, with 2.82 Modified Adverse Events per 10,000 Patient Encounters (Fig. 4) observed over the study period.
Consistent with the development of the EMS triggers, our study excluded several records during the sampling process, including evidence-based care pathways and high-risk/infrequent medications and procedures. Although these cases represent a small proportion of all EMS calls, the potential exists that their exclusion skewed the results of the analysis. The removal of these records was deliberate, given the aim of the tool for use in the high frequency/low risk cases often excluded from regular EMS audit programs. This limitation represents an avenue for further research and evaluation of the EMSTT in a sample where such exclusion criteria have not been applied.
Lastly, the aim of this research was limited to the development of implementation, data collection and reporting processes, and to test the EMSTT using these processes over time. As such, the study did not include tests for association or correlation between trigger items and rates of AEs and/or Harm. Similarly, this study did not include efforts aimed at improving or reducing the impact of the most common triggers. These remain as areas for future research.
The TT methodology has seen considerable success as a quick, reliable and simple, yet comprehensive method of detecting AEs within healthcare. These desirable characteristics have led to the successful development of trigger tools within a variety of disciplines, including general healthcare , surgical care , primary care , intensive care , paediatric care  as well as pharmacy  and laboratory services . However, integrating such a system into regular operation can be a complex task and should be viewed as equally important as the content of the tool itself [11, 12]. Despite the robust planning and processes put in place, several issues became apparent that hindered data collection for several months, requiring constant intervention and refinement to correct, and included:
Communication of the process and operational definitions emerged as a central component both internally within the project team, and externally with all departments the project could potentially impact.
Dedicated, protected time for each step within the process was essential to complete the required reviews. When the time required to complete each review round was specifically scheduled, review rounds were quicker and easier to complete.
The review team should be kept to a small, dedicated number of staff, to allow for consistency and familiarity of the tool and processes to develop over time. This proved to be beneficial from a resources and time investment vs. output point of view.
The importance of a contextually relevant and appropriate AE/Harm classification system became increasingly apparent throughout the analysis. As comprehensive as the NCC MERP system is, the nuances of the EMS environment made application of this system difficult. The fit for purpose AE Severity Index, while more subjective, provided a simple and easy to implement alternative that warrants further examination.
Documentation adequacy and legibility created difficulties in separating deficiencies in care from deficiencies in documentation. For such a system to be successful, it is essential that a review of documentation quality be carried out in parallel, or that a service have sufficient recourse to minimise the effect of poor documentation. The increased uptake of electronic medical records may limit the effects of some of these issues.
Given the above-mentioned period of transition towards an electronic PCR, the topic of automation was discussed amongst the project team on several occasions. As the project progressed, it became increasingly apparent that the iterative manual review of cases over time added to the learning experience in trying to understand the interaction of triggers to adverse events and harm. It additionally offered insight into future potential trigger items, knowledge that would have otherwise been lost had the process become automated.
This study was amongst the first to test and analyse a trigger tool and its operational processes specific to the EMS setting. The incidence of trigger items observed in this analysis echoed the observations reported in the development of the EMS triggers, with the significant majority of trigger positive cases comprising the same three triggers. The incidence of AEs and Harm identified was relatively low, including the modified measures, when compared to the development of the EMS triggers. However, these differences could potentially be attributed to differences in sampling, with the focus on small consecutive samples in this analysis, compared to the single large sample focus with the initial development. The focus for this study was modelled on the sampling approach used by the GTT, which has shown reliable predictability compared with larger samples . Nonetheless, the greater ease and shorter duration of time required to assess EMS PCRs, using the EMSTT, remains an avenue for further study employing larger samples.
No other peer-reviewed research exists examining the application of a TT over time, within the EMS setting. As such, there is no benchmark with which to compare the results observed in this study. Despite this, it stands to reason that variation in service models found in EMS around the world could have the potential to impact the results of the application of the EMSTT in other settings. This represents a significant opportunity for future research, to better understand the prevalence and incidence of AEs within EMS. While many in-hospital TTs exist, variation in patient cohorts, area of service, length of exposure to the healthcare service studied and varying outcome measures make comparison with the results of this study impossible.
TTs represent a potential strategy towards the successful identification of the incidence of AE and harm within healthcare. With a better understanding of the case types and causes of AEs and Harm, targeted improvement projects can be designed and implemented to advance quality and patient safety. This study is amongst the first to test and analyse an EMS specific TT over time, in order to measure the rate of AE/Harm in this setting. With sufficient focus on implementation and data collection, and the inclusion of a contextually relevant classification system, the EMSTT represents a potentially successful strategy towards identifying the rate of AE/Harm within EMS.
Emergency medical services
Emergency Medical Services Trigger Tool
Global Trigger Tool
Hamad Medical Corporation Ambulance Service
Institute for Healthcare Improvement
- NCC MERP:
National Coordinating Council for Medication Error Reporting and Prevention
Patient Care Record
Committee on Quality of Health Care in America I of M. To err is human: building a safer health system. 2000.
Committee on Quality of Health Care in America I of M. Crossing the quality chasm: A New Health System for the 21st Century (Online). Washington, DC: National Academies Press; 2001. Available from: http://www.nap.edu/catalog/10027/crossing-the-quality-chasm-a-new-health system-for-the.
Thomas EJ, L a P. Measuring errors and adverse events in health care. J Gen Intern Med. 2003;18:61–7.
Payne TH, Savarino J, Marshall R, Hoey CT. Use of a clinical event monitor to prevent and detect medication errors. Proc AMIA Symp. 2000:640–4.
Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform. 2003;36(1–2):131–43.
Naessens JM, Naessens JM, Campbell CR, Campbell CR, Huddleston JM, Huddleston JM, et al. A comparison of hospital adverse events identi ed by three widely used detection methods. Int J Qual Heal Care. 2009;21(4):301–7.
NHS Wales. Tools for improvement: how to use trigger tools [internet]. Wales; 2012. Available from: http://www.1000livesplus.wales.nhs.uk/sitesplus/documents/1011/T4I%20%284%29%20How%20to%20use%20Trigger%20Tools%20%28Feb%202011%29%20Web.pdf
Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Qual Saf. Health Care. 2003;12(2):39–46.
Brown S, Black K, Mrochek S, Wood A, Bess T, Cobb J, et al. RADARx: recognizing, assessing, and documenting adverse Rx events. Proc AMIA Symp. 2000:101–5.
Howard I, Bowen J, Al Shaikh L, Mate K, Owen R, Williams D. Development of a trigger tool to identify adverse events and harm in emergency medical services. Emerg Med J (Online). 2017;0:1–7. https://doi.org/10.1136/emermed-2016-205746.
Griffin F, Resar R. IHI global trigger tool for measuring adverse events [internet]. IHI Innovation Series white paper. 2007; Available from: http://www.ihi.org/resources/Pages/IHIWhitePapers/IHIGlobalTriggerToolWhitePaper.aspx.
Adler L, Denham CR, McKeever M, Purinton R, Guilloteau F, Moorhead JD, Resar R. Global trigger tool: implementation basics. J Patient Saf. 2008;4(4):245–9.
Hartwig SC, Denger SD, Schneider PJ. Severity-indexed, incident report-based medication error-reporting program. Am J Hosp Pharm. 1991;48:2611–6.
Patterson PD, Weaver MD, Abebe K, Martin-Gill C, Roth RN, Suyama J, et al. Identification of adverse events in ground transport emergency medical services. Am J Med Qual. 2012;27:139–46.
Provost LP, Murray SK. The health care data guide: learning from data for improvement. 1st ed. San Francisco, CA: John Wiley & Sons; 2011.
Berwick DM. Controlling variation in health care: a consultation from Walter Shewhart. Med Care. 1991;29(12):1212–25.
Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care. 2003;12:458–64.
Landis RJ, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33:363–74.
Haraden C. Personal. Communication. Mar 2017.
Griffin F, Classen DC. Detection of adverse events in surgical patients using the trigger tool approach. Qual Saf. Health Care. 2008;17(4):253–8.
de Wet C, Bowie P. The preliminary development and testing of a global trigger tool to detect error and patient harm in primary-care records. Postgrad Med J. 2009;85(1002):176–80.
Resar RK, Rozich JD, Simmonds T, Haraden CR. A trigger tool to identify adverse events in the intensive care unit. Jt Comm J Qual Patient Saf. 2006;32(10):585–90.
Matlow AG, Cronin CMG, Flintoft V, Nijssen-Jordan C, Fleming M, Brady-Fryer B, et al. Description of the development and validation of the Canadian Paediatric trigger tool. BMJ Qual Saf. 2011;20(5):416–23.
Ganachari M, Wadhwa T, Walli S, Dhisha K, Aggarwal A. Trigger tools for monitoring and reporting of adverse Frug reactions: a scientific tool for efficient reporting. Open Access Sci Reports. 2013;2(4):1–5.
Brenner S, Detz A, López A, Horton C, Sarkar U. Signal and noise: applying a laboratory trigger tool to identify adverse drug events among primary care patients. BMJ Qual Saf. 2012;21(8):670–5.
Availability of data and materials
Please contact author for data requests.
Ethics approval and consent to participate
Ethical approval was granted by the Hamad Medical Corporation Academic Health System Institutional Review Board.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Howard, I., Pillay, B., Castle, N. et al. Application of the emergency medical services trigger tool to measure adverse events in prehospital emergency care: a time series analysis. BMC Emerg Med 18, 47 (2018). https://doi.org/10.1186/s12873-018-0195-0
- Emergency medical services
- Trigger tool
- Patient safety
- Adverse events, harm