Skip to main content

Evaluation of cardiopulmonary resuscitation quality during the pandemic of COVID-19

Abstract

Background

Cardiopulmonary resuscitation (CPR) is an important technique of first aid. It is necessary to be popularized. Large-scale offline training has been affected after the outbreak of Coronavirus disease 2019 (COVID-19). Online training will be the future trend, but the quality of online assessment is unclear. This study aims to compare online and offline evaluations of CPR quality using digital simulator and specialist scoring methods.

Methods

Forty-eight out of 108 contestants who participated in the second Chinese National CPR Skill Competition held in 2020 were included in this study. The competition comprised two stages. In the preliminary online competition, the contestants practiced on the digital simulator while the specialist teams scored live videos. The final competition was held offline, and consisted of live simulator scoring and specialist scoring. The grades of the simulator and specialists in different stages were compared.

Results

There was no statistical significance for simulator grades between online and offline competition(37.7 ± 2.0 vs. 36.4 ± 3.4, p = 0.169). For specialists’ grades, the video scores were lower than live scores (55.0 ± 1.4 vs. 57.2 ± 1.7, p < 0.001).

Conclusion

Simulator scoring provided better reliability than specialist scoring in the online evaluation of CPR quality. However, the simulator could only collect quantified data. Specialist scoring is necessary in conjunction with online tests to provide a comprehensive evaluation. A complete and standardized CPR quality evaluation system can be established by combining simulator and specialist contributions.

Peer Review reports

Introduction

Cardiac arrest (CA) is one of the major causes of death and an increasing health concern worldwide [1]. In one year, 356,500 Americans and 115,600 Japanese suffered out-of-hospital CA [2, 3]. The annual number of sudden deaths from CA in China was estimated from a regional survey at approximately 540,000 [4]. Cardiopulmonary resuscitation (CPR) is widely recognized as a central component of first-aid for CA [5]. According to the latest international guidelines, high-quality CPR with rapid defibrillation is a focal point in resuscitating adult and pediatric victims of CA [6]. The quality of CPR directly affects the survival rate and neurological outcome of CA resuscitation [7, 8]. If effective CPR is not started within four minutes after CA, the cerebral ischemic and hypoxic damage will be irreversible [9]. Although CPR was been included in basic first-aid training for decades, effective CPR administered by bystanders remains infrequent [10]. Universal CPR training still plays an essential role in the promotion of public health [11].

Coronavirus disease 2019 (COVID-19) is a collection of syndromes caused by SARS-CoV-2. Since December 2019, COVID-19 cases have been identified in all countries [12]. The World Health Organization has declared it a pandemic disease [12, 13]. The pandemic and the consequent lockdowns introduced major changes in behavior regarding work, cooperation, and learning [14]. Regular training is important in increasing willingness and confidence when performing CPR [15, 16]. The offline CPR courses were paused because of lock-down strategies. Among medical professionals, the CPR quality is known to decrease within only three months after training [5, 17]. Traditional CPR training combines face-to-face courses with practice on simulators. Trainers make real-time evaluations of quality and correct the mistakes of trainees. Online training has been evaluated as a potential substitute. Online remote qualifications with Internet and smart CPR simulators have been available for years [18, 19]. A randomized trial confirmed the practicality of offline objective CPR skill assessment by a simulator [20]. There remain limitations such as high cost, lack of supervision, fraud, and cheating in evaluations [21].

To evaluate the quality of online CPR training, a smart simulator and video scoring by remote specialists are both technically accessible. So far, few studies have described the effectiveness of online evaluation using the two techniques. In this study, we analyzed data obtained from National CPR Skill Competition held in China, to compare online (via video) and offline (face-to-face) evaluation of CPR quality using digital simulators and specialist scoring methods.

Methods

Participants and data

All contestants in the Second Chinese National CPR Skill Competition were included in this observational study. The Competition was hosted by the Chinese Society of Cardiothoracic and Vascular Anesthesiology (CSCVA). The contestants were medical professionals who received regular CPR training. They participated online in the preliminary competition in 54 separate standardized exam rooms around China on October 24th, 2020. Each contestant performed 5 rounds of standard CPR and one automated external defibrillator (AED) defibrillation procedure on a Laerdal Intelligent CPR simulator (174–01260, Resuscitation Anne QCPR, Laerdal, Norway). The sensors and scoring system attached to the simulator recorded real-time data and calculated the total score automatically. The scoring system criteria are presented in Table S1 of the Supplementary material. Simultaneously, a group of five CSCVA-certified CPR specialists monitored contestants with two Internet cameras and scored their performance on the criteria in Table S2 of the Supplementary material. The offline final competition (face-to-face competition) was held in Guangzhou on November 13th, 2020. Finals participants received no retraining after the preliminary competition. In an offline exam room, they performed the same procedures on the same model of the simulator, and the same specialist group observed and scored their performance on site. The competition flow can be seen in Fig. 1.

Fig. 1
figure 1

Competition flow chart

Statistical analysis

Data are presented as mean ± standard deviation. Counts are by frequency and composition ratio. Repeated measures, paired sample T-test, analysis of variance, and non-parametric test were used to compare the differences in scores of the same players in the preliminary and the finals. P < 0.05 was considered statistically significant. All statistical analyses were performed using SPSS 25.0 software.

Results

There were 108 contestants in the preliminary competition and 48 of them entered the final round. Among the 48 analyzed participants, 19 (40%) were men and 29 (60%) were women. The average age was 30 ± 4.9 years old, ranging from 22 to 44 years. For simulator scoring, the standard derivation of the online competition was 2.0 and offline was 3.4. The Kruskal-Wallis test showed no statistical significance between online and offline scoring (p = 0.169) (Table 1). For specialist scoring, the standard derivation of the online competition was 1.4 and offline was 1.7. The online and offline scores were statistically different with a paired sample t-test (p < 0.001) (Table 1). Such variation in data stability could also be seen in Fig. 2.

Table 1 The difference in scoring between evaluation methods
Fig. 2
figure 2

Box plots of participants’ scores. Simulator scoring of practice in preliminary and final content. Specialist scoring of practice in preliminary and final content

Discussion

The present study analyzed the scores of the contestants in a hybrid CPR skill competition comprising online and offline components. We found no statistical difference between online and offline evaluations of CPR quality based on sensors and programs on a digital simulator. The specialist scoring is significantly different between online and offline evaluations. The offline contestants tended to get higher scores from the specialist scoring system. However, we found that the simulator scores had higher standard derivation, this may be related to the state of competitors on the day and simulators will give higher resolution scoring. Also, this will reveal that simulator scoring may be more objective than specialist scoring.

The most critical maneuvers affecting a patient’s prognosis in CPR are compression depth and frequency, both of which are quantifiable indicators [5]. An instrumented direct feedback device measures compression rate, depth, hand position, recoil, and chest compression fraction and provides real-time audio or visual feedback (or both) on these critical CPR skills [22]. Training for chest compressions based on the use of real-time feedback software (Laerdal QCPR) guided by an instructor is superior to instructor-guided feedback training in terms of overall chest compression technical skill acquisition [11]. Therefore, the CPR quality evaluation system is effective for simulator scoring. However, the simulator is too sensitive: a slight change in the subject’s action may lead to fluctuations in the results. So it is not enough solely rely on machines.

In our study, practice quality evaluation in both online and offline modes did not affect the objective results obtained by the simulator evaluation. Therefore, it is essential for the evaluation of the quality of CPR on an objective basis. The more common specialist scoring approach also is more subjective. Even if the average score of five specialists is adopted, differences remain. Therefore, the subjective scores obtained by the judges should be considered a supplement to the objective scores, such as compression depth and frequency.

Medical simulations have great promise to provide training at less cost and without risk to patients. However, these advantages are not sufficient to conclude effectiveness [23]. A study showed that simulator assessment can be sufficient but they also suggest a combination of assessment tools will be much better [24]. This study analyzing two competitions with the same contestants found the offline specialist scoring to be significantly higher than the online specialist scoring. Explanations could reflect on both players and the specialists: Players sensed the offline finals as more important than online, making the onsite competition more urgent, preparation more extensive, and the competition mode more familiar. The onsite performances were more stable than in the preliminary round, and the results are slightly higher. Specialists applied the same scoring standards for the preliminary and final rounds. And the judges were all senior professors. Although there are three screens for online viewing, the details may not have been clear enough. Plus, the atmosphere of the finals and the neatness of the players’ clothing could cause specialists to subjectively increase emotional points.

This is consistent with the conclusion of the article published by Camilla Hansen and others in 2019. They believe that the basic life support (BLS) certified instructors still have a poor assessment of the quality of CPR, and the compression depth and artificial respiration are not clear [25]. Machines can ensure quality through a quantifiable index. Specialists can objectively integrate many factors to ensure the stability of the score. We need to combine these and establish a new quality evaluation system for online training.

The results of this article suggest that the current CPR quality evaluation system should not depend only on specialist scoring, and that differences in competition scenes will significantly affect the specialist scoring. Human judgments are susceptible to error due to a host of factors, from fatigue to various biases of judges [26]. Specialist scoring can be used as a supplement to the simulator scoring, and indicators that cannot be quantified and monitored, such as whether the practice posture is standardized, whether the arm is completely perpendicular to the ground when pressing, etc. Due to the high price of current simulators with evaluation functions, it is difficult to train enough people on CPR. Thus, adopting new approaches, technology, and further research are necessary.

CPR is the most effective rescue method for patients with CA. The quality of CPR is low during both in-hospital and out-of-hospital rescue [27]. The domestic penetration rate of CPR training is lower in China than in other countries [28]. As populations are, the prevalence of cardiovascular diseases is increasing. To achieve a rapid increase in the penetration rate in the short term and improve the quality of CPR, online training with feedback devices is the most apparent path.

The American Heart Association(AHA) statement on resuscitation education science emphasizes the importance of using objective data to improve BLS and advanced life support skills. The 2015 European CPR Guidelines also recommend the use of feedback devices to improve the quality of CPR [11]. However, feedback devices may be prohibitively expensive and not available in all situations. There is currently no other objective method to assess the quality of CPR [25].

Studies have shown that training with assistive devices can improve the quality of CPR [20]. Online training will be an inevitable trend in the post-epidemic era. CPR requires regular retaining to ensure the quality of operation. Online training can facilitate this. The AHA guidelines also suggest that continuous enhancement of this skill in the short term can improve the quality of practice [17]. Even doctors who have received multiple pieces of training should retrain regularly to improve the quality of their practice and ensure clinical safety.

This study has several limitations. Because of the influence of COVID-19, it is not possible to organize large-scale activities for comparison the sample size was lacking. All the participants come from different medical institutions and their educational backgrounds differed, which may have influenced comparisons. As this study only involved medical staff, the results apply only to that occupation.

Conclusion

Online CPR quality evaluated by digital simulators can better reflect the actual performance level of trainees, and be consistent with the traditional offline evaluation. Simulators can be used as an important means of monitoring training in online CPR training. Standardized CPR quality evaluation should advance beyond the traditional evaluation model with a quality evaluation system established on objective results obtained after the quantitative evaluation of the cardiopulmonary resuscitation simulator and supplemented by the subjective results of evaluators.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Abbreviations

CPR:

Cardiopulmonary Resuscitation

COVID-19:

 Coronavirus disease 2019

CA:

Cardiac arrest

CSCVA:

Chinese Society of Cardiothoracic and Vascular Anesthesiology

AED:

Automated external defibrillator

BLS:

Basic life support

AHA:

American Heart Association

References

  1. Nakahara S, Tomio J, Ichikawa M, Nakamura F, Nishida M, Takahashi H, et al. Association of Bystander Interventions with neurologically intact survival among patients with bystander-witnessed out-of-hospital cardiac arrest in Japan. JAMA. 2015;314(3):247–54.

    Article  CAS  Google Scholar 

  2. Suematsu Y, Zhang B, Kuwano T, Sako H, Ogawa M, Yonemoto N, et al. Citizen bystander-patient relationship and 1-month outcomes after out-of-hospital cardiac arrest of cardiac origin from the All-Japan Utstein Registry: a prospective, nationwide, population-based, observational study. BMJ open. 2019;9(7):e024715.

    Article  Google Scholar 

  3. McNally B, Robb R, Mehta M, Vellano K, Valderrama AL, Yoon PW, et al. Out-of-hospital cardiac arrest surveillance –- Cardiac Arrest Registry to Enhance Survival (CARES), United States, October 1, 2005–December 31, 2010. MMWR Surveill Summ. 2011;60(8):1–19.

    Google Scholar 

  4. Hua W, Zhang LF, Wu YF, Liu XQ, Guo DS, Zhou HL, et al. Incidence of sudden cardiac death in China: analysis of 4 regional populations. J Am Coll Cardiol. 2009;54(12):1110–8.

    Article  Google Scholar 

  5. Pritchard J, Roberge J, Bacani J, Welsford M, Mondoux S. Implementation of chest Compression Feedback Technology to improve the quality of cardiopulmonary resuscitation in the Emergency Department: a Quality Initiative Test-of-change study. Cureus. 2019;11(8):e5523.

    Google Scholar 

  6. Craig-Brangan KJ, Day MP. Update. 2017/2018 AHA BLS, ACLS, and PALS guidelines. Nursing. 2019;49(2):46–9.

    Article  Google Scholar 

  7. Lin L, Ni S, Cheng J, Zhang Z, Zeng R, Jin X, et al. Effect of synchronous online vs. face-to-face cardiopulmonary resuscitation training on chest compression quality: a pilot randomized manikin study. Am J Emerg Med. 2021;50:80–4.

    Article  Google Scholar 

  8. Nabecker S, Theodorou M, Huwendiek S, Kasper N, Greif R. Out-of-hospital cardiac arrest: comparing organised groups to individual first responders: a qualitative focus group study. Eur J Anaesthesiol. 2021;38(10):1096–104.

    Article  Google Scholar 

  9. Yuksen C, Sawatmongkornkul S, Tuangsirisup J, Sawanyawisuth K, Sittichanbuncha Y. The CPR outcomes of online medical video instruction versus on-scene medical instruction using simulated cardiac arrest stations. BMC Emerg Med. 2016;16(1):25.

    Article  Google Scholar 

  10. Leong BS. Bystander CPR and survival. Singapore Med J. 2011;52(8):573–5.

    CAS  Google Scholar 

  11. Cortegiani A, Russotto V, Montalto F, Iozzo P, Meschis R, Pugliesi M, et al. Use of a Real-Time Training Software (Laerdal QCPR®) compared to instructor-based feedback for high-quality chest Compressions Acquisition in secondary School students: a Randomized Trial. PLoS ONE. 2017;12(1):e0169591.

    Article  Google Scholar 

  12. Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet (London England). 2020;395(10223):497–506.

    Article  CAS  Google Scholar 

  13. Sockrider M, Krishnan V. Socializing during the COVID-19 pandemic. Am J Respir Crit Care Med. 2021;203(1):P3-p4.

    Article  Google Scholar 

  14. Secundo G, Mele G, Vecchio PD, Elia G, Margherita A, Ndou V. Threat or opportunity? A case study of digital-enabled redesign of entrepreneurship education in the COVID-19 emergency. Technol Forecast Soc Chang. 2021;166:120565.

    Article  Google Scholar 

  15. Kim JW, Lee JH, Lee KR, Hong DY, Baek KJ, Park SO. Improvement in trainees’ attitude and resuscitation quality with repeated cardiopulmonary resuscitation Training: cross-sectional Simulation Study. Simul healthcare: J Soc Simul Healthc. 2016;11(4):250–6.

    Article  Google Scholar 

  16. Więch P, Sałacińska I, Muster M, Bazaliński D, Kucaba G, Fąfara A, et al. Use of selected Telemedicine Tools in Monitoring Quality of In-Hospital cardiopulmonary resuscitation: a prospective Observational Pilot Simulation Study. Med Sci monitor: Int Med J experimental Clin Res. 2019;25:2520–6.

    Article  Google Scholar 

  17. Chang TP, Raymond T, Dewan M, MacKinnon R, Whitfill T, Harwayne-Gidansky I, et al. The effect of an International competitive leaderboard on self-motivated simulation-based CPR practice among healthcare professionals: a randomized control trial. Resuscitation. 2019;138:273–81.

    Article  Google Scholar 

  18. Peberdy MA, Silver A, Ornato JP. Effect of caregiver gender, age, and feedback prompts on chest compression rate and depth. Resuscitation. 2009;80(10):1169–74.

    Article  Google Scholar 

  19. Pozner CN, Almozlino A, Elmer J, Poole S, McNamara D, Barash D. Cardiopulmonary resuscitation feedback improves the quality of chest compression provided by hospital health care professionals. Am J Emerg Med. 2011;29(6):618–25.

    Article  Google Scholar 

  20. Tanaka S, Hara T, Tsukigase K, Sagisaka R, Myklebust H, Birkenes TS, et al. A pilot study of Practice while Watch based 50 min school quality cardiopulmonary resuscitation classroom training: a cluster randomized control trial. Acute Med Surg. 2020;7(1):e455.

    Article  Google Scholar 

  21. Bell BS, Federman JE. E-Learning in postsecondary education. The Future of children. 2013;23(1):165–85.

    Article  Google Scholar 

  22. Association AH. Official directive of American Heart Association (AHA) requirement on use of feedback devices in adult CPR training courses 2019. Available from: https://ahainstructornetwork.americanheart.org/idc/groups/ahaecc-public/@wcm/@ecc/documents/downloadable/ucm_495639.pdf?fbclid=IwAR0m89xIk7rePW5BMhR-o3ddxeZ710_. January 1st 2022.

  23. Bewley WL, O’Neil HF. Evaluation of medical simulations. Mil Med. 2013;178(10 Suppl):64–75.

    Article  Google Scholar 

  24. Dubin AK, Smith R, Julian D, Tanaka A, Mattingly P. A comparison of robotic Simulation performance on basic virtual reality skills: Simulator Subjective Versus Objective Assessment Tools. J Minim Invasive Gynecol. 2017;24(7):1184–9.

    Article  Google Scholar 

  25. Hansen C, Bang C, Stærk M, Krogh K, Løfgren B. Certified basic life support instructors identify improper cardiopulmonary resuscitation skills poorly: instructor assessments versus resuscitation manikin data. Simul Healthc. 2019;14(5):281–6.

    Article  Google Scholar 

  26. Mazurova E, Standaert W, Penttinen E, Tan FTC. Paradoxical tensions related to AI-Powered evaluation systems in competitive sports. Inf Sys Front. 2021;24(3):897–922.

  27. Brennan EE, McGraw RC, Brooks SC. Accuracy of instructor assessment of chest compression quality during simulated resuscitation. Cjem. 2016;18(4):276–82.

    Article  Google Scholar 

  28. Tanaka S, Tsukigase K, Hara T, Sagisaka R, Myklebust H, Birkenes TS, et al. Effect of real-time visual feedback device ‘Quality Cardiopulmonary Resuscitation (QCPR) Classroom’ with a metronome sound on layperson CPR training in Japan: a cluster randomized control trial. BMJ open. 2019;9(6):e026140.

    Article  Google Scholar 

Download references

Acknowledgements

We thank the staff of Chinese Society of Cardiothoracic and Vascular Anesthesiology for their generous help in data collection.

Statements

The results and methods were reported following the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines.

Funding

There is no funding used in this study.

Author information

Authors and Affiliations

Authors

Contributions

Yang Yu: Conceptualization, Methodology, Formal analysis, Investigation, Writing-original draft, Project administration. Xiaojie Liu: Validation, Writing-review & editing, Lijuan Wang: Formal analysis, Data curation, Yuchen Gao: Investigation, Writing-review & editing, Yao Ding: Resources, Visualization. Hushan Ao: Conceptualization, Methodology, Writing-review & editing, Supervision. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Hushan Ao.

Ethics declarations

Ethics approval and consent to participate

This study was conducted in accordance with the Declaration of Helsinki. All included contestants signed an informed consent form, and this study was approved by ethical committees of Fuwai Hospital, Chinese Academy of Medical Sciences, the ethics approval number is 2021 − 1625.

Consent for publication

This paper doesn’t include details, images, or videos relating to a person.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table S1.

The criteria of scoring system in simulator. Table S2. The criteria of specialist scoring.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yu, Y., Liu, X., Wang, L. et al. Evaluation of cardiopulmonary resuscitation quality during the pandemic of COVID-19. BMC Emerg Med 22, 193 (2022). https://doi.org/10.1186/s12873-022-00754-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12873-022-00754-x

Keywords

  • Cardiopulmonary resuscitation
  • Simulator
  • Evaluation
  • Online