Reporting of unintended events in an intensive care unit: comparison between staff and observer

Background In order to identify relevant targets for change, it is essential to know the reliability of incident staff reporting. The aim of this study is to compare the incidence and type of unintended events (UE) reported by facilitated Intensive Care Unit (ICU) staff with those recorded concurrently by an observer. Methods The study is a prospective data collection performed in two 4-bed multidisciplinary ICUs of a teaching hospital. The format of the UE reporting system was voluntary, facilitated and not necessarily anonymous, and used a structured form with a predetermined list of items. UEs were reported by ICU staff over a period of 4 weeks. The reporting incidence during the first fourteen days was compared with that during the second fourteen. During morning shifts in the second fourteen days, one observer in each ICU recorded any UE seen. The staff was not aware of the observers' study. The incidence of UEs reported by staff was compared with that recorded by the observers. Results The staff reported 36 UEs in the first fourteen days and 31 in the second.. The incidence of UE detection during morning shifts was significantly higher than during afternoon or night shifts (p < 0.001). Considering only working day morning shifts, the rate of UE reporting by the staff per 100 patient days was 26.9 (CI 95% 16.9–37.0) in the first fourteen day period and 20.3 (CI 95% 10.3–30.4) in the second. The rate of UE detection by the observers was 53.1 per 100 patient days (CI 95% 40.6–65.6), significantly higher (p < 0.001) than that reported concurrently by the staff. There was excellent agreement between staff and observers about the severity of the UEs recorded (Intraclass Correlation Coefficient 0.869). The observers recorded mainly UEs involving Airway/mechanical ventilation and Patient management, and the staff Catheter/Drain/Probe and Medication errors (p = 0.025). Conclusion UE incidence is strongly underreported by staff in comparison with observers. Also the types of UEs reported are different. Invaluable information about incidents in ICU can be obtained in a few days by observer monitoring.


Background
Voluntary and anonymous Critical Incident Reporting (CIR) in Intensive Care Units (ICUs) is a technique for collecting information about any unintended event or outcome that reduced or could have reduced the safety margin for the patient. CIR has been used in single adult [1][2][3][4][5] as well as neonatal-pediatric [6] ICUs, and implemented at a national level in Australia [7][8][9].
It has been claimed that the rate of critical incidents could be a marker of the quality of care [5]. Unfortunately, CIR does not provide an objective numerator and there is no unequivocal denominator. Taking patient days as denominator, Bracco and co-workers [4] reported 777 critical incidents (241 of them were human errors) in 2810 patient days, giving a rate of 27.4 critical incidents (8.6 human errors) per 100 patient days, while Flaatten and Hevroy [10] found 2.7 errors per 100 patient days and, more recently, Osmon and co-workers [11] reported 8.93 medical errors per 100 ICU days. On the other hand, a pharmacist identified 187 medication administration errors in 851 patients [12], and an observer detected 132 medication errors in 88 patient days of observation [13]. Moreover, it has been shown that ICU staff reported only 61% of the errors detected by an external observer [14]. In some cases, such differences may reflect different definitions, but in others [3][4][5][6] they suggest that the real size of the patient safety problem in ICU is unknown.
The ultimate goal of incident reporting is to implement strategies to prevent recurrence [15]. Therefore, to ensure that relevant targets for change are recognized, it is essential to know the reliability of staff reporting.
The aim of this study is to compare the incidence and type of unintended events reported by facilitated ICU staff with those recorded concurrently by an observer.

Design
The study lasted for 28 days in 2003, from 24 November (7.00 am) to 22 December (7.00 am). Staff reporting of UEs on structured forms was active throughout the period. The UEs reported by the staff during the first fourteen days were compared with those reported during the second fourteen to detect any change in staff reporting.
During morning shifts (7.00 am-14.00 pm) in the second fourteen days (weekends and holidays were excluded), one observer in each ICU who had no other duties recorded any UE seen. The observations were performed concurrently in two ICUs and compared with the incidence of UE reporting by staff over the same period of time, to measure the reliability of spontaneous reporting.
The Ethics Committee of the Hospital approved the study.

Setting
The study was performed in two four-bed multidisciplinary ICUs located in different structures of the same 916bed teaching hospital. Each ICU consists of a wide room with four beds allowing direct patient observation, with sliding curtains between beds.
Both ICUs have the same nurse-to-patient ratio, which is 1:2 (excluding 1 daytime unit sister per each ICU, with organizational and administrative tasks), and three nurse shifts: morning (7.00-14.00), afternoon (14.00-22.00) and night (22.00-7.00). In the ICUs, medical care is provided by specialists in anaesthesia and intensive care according to national rules. The ICUs have one overall medical director. A full time specialist and a resident doctor manage each ICU in the daytime (8.00-20.00), while a specialist and a resident doctor cover both ICUs at night (20.00-8.00).

Procedure
Before the study was initiated, several meetings with the research team were held to familiarize the ICU staff with the concept of CIR, to describe the non-punitive nature of the study, and to develop a structured form with a predetermined list of UEs for data collection. The form was tested in both ICUs, modified according to staff comments (excluding items never used and adding space for possible items not previously considered) and then implemented. The list of the items in the structured reporting form is given in the Table 1. The staff was instructed to report any UE irrespective of whether it was on the predetermined list, and to use the appropriate space in the form if the item was not listed.
During the period of the study, the Director of the Unit enhanced reporting by emphasizing its relevance during ward rounds, encouraging staff to fill in the form immediately after discovering any UE, and discussing relevant (minor or serious) reports weekly with the whole ICU staff. Reporting was facilitated by the cooperative attitude of all the staff and by allowing but not requiring anonymity. The format of UE reporting system was voluntary, non-punitive, facilitated and not necessarily anonymous.
Blank forms were available on the nursing desk and completed ones were stored in a free deposit box, which was emptied every evening by those responsible for the study, who first analysed the reports. A poster with guidelines for reporting, definitions of UEs, some examples and the name of the person responsible for the research was displayed on the wall next to the forms.
During morning shifts in the second fourteen day period of the study, two residents attending the Specialist School in Anaesthesia and Intensive Care of the University (one in each ICU) acted as observers. They had already performed a one-year training period in general ICU and, before the beginning of the study, they had studied CRI methodology, analyzed precise definitions of variables in the literature and received instruction, with examples, about the definitions of UEs used in the present study. The observers, like the staff, were instructed to report any UE irrespective of whether it was in the predefined list, and to use the appropriate space in the form if the item was not listed. The observers used the same rules and structured forms as the staff, but they filled in a new form for each UE in a separate room and retained it in a separate box until the end of the study. The staff was not aware of the observers' study; the presence of the observers was explained by the collection of data for a different study.

Definitions
For the purpose of the study, any unintended event (UE) that reduced or could have reduced the safety margin for the patient while in ICU was considered. UEs occurring during transport or in other areas of the hospital were not considered. The following information, collected for each UE (items listed in the Table 1), was analyzed: date and shift of detection, reporter's qualification (nurse or physician), type and severity. The type of UE was categorized as follows [8]: Airway/mechanical ventilation, Catheter/ Drain/Probe, Medication error, Patient management, Unit management, and Other (with space for details). UE severity was classified according to the reporter's opinion of the outcome and defined as follows: • intercepted and spontaneously rectified by the staff before the patient could be affected (for example: a doctor prescribed a wrong dose of a drug, the nurse realized that it was a mistake and informed the doctor, who corrected the error so that the patient received the correct dose); • self-resolving without specific treatment at the time of the event without harm to the patient (for example: accidental removal of a nasogastric tube that was no longer necessary, or alarms turned off after the end of nursing activities or physician round); • minor at the time of the event but requiring transient increase in surveillance or adjustment of treatment (for example: unplanned intubation after planned extubation, without major physiological complications);

Severity of the unintended event
Intercepted by the staff Self resolving Minor Serious • serious, i.e. life-threatening or responsible for an increased length of hospitalization.

Data analysis
Statistical analysis was carried out using a software package (SigmaStat 2.0) and p values less than 0.05 were considered significant. Proportions were reported with 95% Confidence Intervals (CI) [16] and compared by a z test.
For the purpose of the study, the 4-week period was split into two intervals of fourteen days: the first (A) included 10 working and 4 non-working (weekend) days, and the second (B) included 9 working and 5 non-working days (4 weekend days and 1 national holiday). The following comparisons were performed: 1) between UEs reported by the staff during periods A and B, to exclude any effect of the presence of the observer; 2) between number and type of UEs reported by staff and observers during period B, to measure the reliability of spontaneous reporting; 3) between the severity scores given by staff and observer to the same UE, using an Intraclass Correlation Coefficient (ICC). The ICC assesses agreement, ranging from 0 (no agreement) to 1 (scores identical). Values higher than 0.80 indicate good agreement.

Patient characteristics
During the period of study 31 patients were admitted to or already present in the ICU. The median age was 68 y (range 28-92), median duration of mechanical ventilation was 3 days (range 0-21), and median length of stay in ICU was 6 days (range 1-43). The median values of SAPS II [17] and APACHE II [18] were 37 (range 7-67) and 15 (range 7-31) respectively. One patient died in the ICU and 10 patients died in hospital. The characteristics of the ICU patients present during periods A and B are shown in table 2.

Bed day occupation
The exact number of bed days occupied (patient days), computed as sum of the hours spent by each patient in the ICU during the period of study divided by 24, was 209 over the whole 4 week period. The numbers of patient days during the working day morning shifts of periods A and B were 78 and 64, respectively; the numbers of bed days available were 80 and 72, respectively; and bed occupancies were 98% and 89%, respectively.

Unintended events reported by the staff
The staff reported 36 UEs in period A and 31 in period B. There was no significant difference in incidence (33 vs 31/ 100 patient days) between the two periods (p = 0.872), or between the number of UEs reported in periods A and B by physicians and nurses (31 and 25 versus 5 and 6, respectively; p = 0.786). Of the 67 UEs reported by the staff, 52 occurred during working and 15 in non-working days, with incidences of 36 and 22/100 patient days, respectively (p = 0.061). Considering the 4 weeks as a whole, the incidence of UEs detected during morning shifts (22/100 patient days) was significantly higher(p < 0.001) than in afternoon (6/100 patient days) or night (4/ 100 patient days) shifts.
Of the 67 UEs reported by staff, 8 were classified as intercepted by the staff (for instance, 5 medication errors in drug prescription were rectified before administration), 45 as self-resolving, 13 as minor and 1 as serious. The last was a ventricular fibrillation, treated without sequelae. The patient's serum potassium level was low according to a laboratory test on a blood sample taken in the morning. Potassium chloride 40 mEq was prescribed to be added to the parenteral bag. The nurse injected the drug into the bag without stopping the infusion pump. In 1 min, the patient, who was intubated but alert, became agitated, so the physician and the nurse hastened to him. Afterwards, the patient lost consciousness with the EGC monitor showing ventricular fibrillation; defibrillation was performed in less than 30 s.

Unintended events detected by observer
In the working day morning shifts of period B, observers detected 34 UEs in 64 patient days. These UEs were classified as intercepted by the staff (3 cases), self-resolving (25), and minor (6). The types of UEs detected by observers were as follows: 13 were in the Airway/mechanical ventilation category, 3 Catheter/Drain/Probe, 5 Medication errors, 12 Patient management and 1 Unit management. The staff and the observer recorded the same severity in 11 of the 13 UEs that were recorded by both (1 intercepted by the staff, 7 self resolving and 3 minor); of the remainder, the observer scored one UE higher than the staff (minor vs self-resolving) and the other lower (self-resolving vs minor). The ICC for the staff and observer severity ratings of the 13 UEs was 0.869, showing excellent agreement.

Comparison between staff and observer reports
The types of UEs reported by the staff during the whole study period (4 weeks), and those reported during period B (morning shifts in working days) by the staff and the observers, are shown in table 3. Of the UEs reported by observers, 6 out of the 13 Airway/mechanical ventilation events involved turning off ventilator alarms and 5 out of the 12 patient management events involved saturation alarms being switched off. The incidences of the 5 types of UE recorded by the observers were significantly different (p = 0.025) from those recorded by staff over the whole study period. No difference was found in the incidences of the 5 types of UE reported by staff during the 4-week study period or in the working day morning shifts of period B (p = 0.265).

Discussion
This study demonstrates that the rate of spontaneous reporting of UEs by ICU staff is less than half an observer's reporting rate. Therefore, information obtained by CIR markedly underestimates the real UE frequency. Even more important, the type of UEs spontaneously reported by staff is different from that recorded by an observer using the same criteria as the staff. If we had had to prioritize changes on the basis of staff reporting in our setting (Table 3, first column), we would have devoted more attention to the Catheter/Drain/Probe and Medication error categories. In contrast, Airway/mechanical ventilation and Patient management were the most frequent types of UE recorded by observers. Therefore, spontaneous staff reporting does not appear reliable enough to mirror the truth.
Generally, 12% (8 of 67) of UEs reported by the staff and 9% (3 of 34) of those detected by observers were intercepted by the staff, suggesting a measure of the efficiency of control in our setting. Most of the UEs recorded by observers involved latent errors in alarm settings, i.e. potential problems within the system [19]. Surprisingly, staff members were not aware of this kind of problem and only the observer reports allowed us to identify it. In this study, most UEs (83%) were reported by physicians. This could be due to the facilitated format of reporting, i.e. the enhancement of immediate reporting by the attending physicians, with stronger emphasis on the report than on the anonymity; and to the cooperative attitude of all the staff, which allowed physicians to record on the form an event discovered by a nurse. The lack of double reports indicates that nurses and doctors, attending and resident, were informed about UEs already reported; the report forms remained in an open box throughout the day of the event. On the other hand, nurses reported 43% of critical incidents [3] and 59% of medical errors [11]. Moreover, nurses reported 74% of incidents in the Australian Incident Monitoring Study published in 1996 [7] and 49% of the incidents collected in 2003, by facilitated incident monitoring, in an ICU where CIR had been used for more than 5 years [5]. This finding and our results suggest that physician reporting increases when it is facilitated and, possibly, not anonymous.
The fact that the highest frequency of UEs was discovered in morning shifts agrees with the findings of Donchin and coworkers [14], who studied the diurnal distribution of errors, and Frey and coworkers, who analyzed critical incidents in pediatrics [6]. Other authors found the highest incidence of critical incidents [3] or errors [10] in afternoon shifts. Our finding is consistent with the high number of activities performed, especially by physicians, during the morning. Nevertheless, in the present study, the time of UE reporting was taken as the time of UE detection, and we cannot exclude the possibility that some UEs reported in morning shifts had actually occurred during other shifts.
The study has some limitations. The use of a predefined list of items could have induced both staff and observers to pay attention mainly to the listed items, even though space was allowed for additional ones. Therefore, we can not exclude the possibility that other UEs were missed. Nor can we exclude the possibility that the staff reported fewer events in period B than in period A due to a lessening of the initial enthusiasm, although there was no statistically significant difference. However, the lower number of patient days in working day morning shifts during period B (64 vs 78 in period A) could explain the lower number of events reported in terms of both direct (reduced number of patients at risk, fewer procedures) and indirect (reduced nursing workload) effects. Another limitation is that nursing workload during the study period was not measured. We did not investigate interobserver reliability, but agreement in responses to questions concerning adverse event reporting has been shown to be good [20]. Indeed, the staff of the two ICUs reported similar percentages of the events recorded by the observers. This finding supports our general conclusion that the incidence of unintended events is markedly underestimated by spontaneous reporting. Finally, no attempt was made to investigate whether the UEs reported were preventable, considering that it has already been shown that most incidents are preventable [3,5].

Conclusion
Our study demonstrates that the incidence of UEs is markedly underestimated by spontaneous staff reporting in comparison with observers' recording. Even more important, the type of UE reported is different. One final implication of this study, performed in a single centre and over a short period of time, is of general value: invaluable information about incidents in ICU can be obtained in a few days by observer monitoring.