A scoping review of evaluation methods for health emergency preparedness exercises
DOI:
https://doi.org/10.5055/ajdm.2018.0292Keywords:
emergency preparedness exercise, health emergency preparedness, evaluation, drills, tabletop, discussion-based exercise, operation-based exerciseAbstract
Objective: To review and analyze evaluation methods currently utilized in health emergency preparedness exercises (HEPE).
Design: This study, part of a larger scoping review that systematically collected and reviewed published evidence related to the benefits of HEPE, provides a further analysis of the evaluation methods utilized in such exercises. We separately analyzed discussion-based and operation-based exercises according to their purpose. This addresses a methodological limitation related to the poorly understood relationship between the purpose and context in which a specific evaluation method is selected to be used.
Results: In the reviewed 64 studies, a variety of evaluation methods were utilized for HEPE including observations, participants’ survey, and post-exercise debriefs. At present, the selection and use of these methods is not guided by any methodology, but seems rather arbitrary. No specific evaluation methods were isolated for any exercise type.
Conclusions: The purpose of evaluation should guide the selection of evaluation methods for HEPE, and these are not context specific. If evaluation is for accountability purposes, such as to test organizational capability to respond, participant feedback should be collected in addition to objective data on performance in an exercise. Advantages of routinely collecting data from exercise participants to study their reactions (exercise feedback, perceptions, satisfaction with the exercise) and routinely conducting post-exercise debriefs (both hot debrief and cold debrief), are discussed to support evaluation for development or learning purposes in any context.
References
International Standard: Societal security - Guidelines for exercises. ISO22398, 1st ed. September 9, 2013. Reference number ISO 22398:2013 (E). Available at https://www.sis.se/api/document/preview/916630/.
European Centre for Disease Prevention and Control: Handbook on Simulation Exercises in EU Public Health Settings. Solna, Sweden: European Centre for Disease Prevention and Control (ECDC), 2014. Available at https://ecdc.europa.eu/sites/portal/files/media/en/publications/Publications/Simulation-exercise-manual.pdf.
BSI: PD 25666: 2010 Business Continuity Management-Guidance on Exercising and Testing for Continuity and Contingency Programmes. London: BSI, 2010.
Swedish Civil Contingencies Agency: Evaluation of Exercises Handbook. Karlstad, Sweden: Swedish Civil Contingencies Agency, 2011.
Harvard School of Public Health: Public health emergency preparedness exercise evaluation toolkit. September 2013. Available at https://www.massgeneral.org/disaster-medicine/assets/PDFs/Public-Health-Emergency-Preparedness-Exercise-Evaluation-Toolkit.pdf.
Gunns J, Leigh M: Developing and delivering exercises. Emerg Plan College. 2016; 3: 1-29.
Beerens RJJ, Tehler H: Scoping the field of disaster exercise evaluation - A literature overview and analysis. Int J Disaster Risk Reduct. 2016; 19(suppl C): 413-446.
Stufflebeam D, Coryn C: Evaluation Theory, Models and Applications. San Francisco, CA: Jossey-Bass, 2014.
Savoia E, Agboola F, Biddinger PD: A conceptual framework to measure systems’ performance during emergency preparedness exercises. Int J Environ Res Public Health. 2014; 11(9): 9712-9722.
Nelson C, Lurie N, Wasserman J: Assessing public health emergency preparedness: Concepts, tools and challenges. Annu Rev Public Health. 2007; 28: 1-18.
Skryabina E, Reedy G, Amlôt R, et al.: What is the value of health emergency preparedness exercises? A scoping review study. Int J Disaster Risk Reduct. 2017; 21(suppl C): 274-283.
Arksey H, O'Malley L: Scoping studies: Towards a methodological framework. Int J Soc Res Methodol. 2005; 8(1): 19-32.
Ablah E, Scanlon E, Konda K, et al.: A large-scale points-of-dispensing exercise for first responders and first receivers in Nassau County, New York. Biosecur Bioterror. 2010; 8(1): 25-35.
Burns KJ, Robinson K, Lowe EG: Evaluation of responses of an air medical helicopter program during a comprehensive emergency response drill. Air Med J. 2007; 26(3): 139-143.
Callagy P, Woodfall M: Drive-through triage and care system: Lessons learned from a pandemic drill. J Emerg Nurs. 2009; 35(6): 588-593.
Miguel AS, Gois J, Silva J: Study on workers’ evacuation in an industrial company. Saf Sci. 2010; 48(8): 1050-1053.
Pennell PA, Antenucci AJ, Brennan LE, et al.: Functional assessment of hospital laboratory packaging and shipping preparedness in New York State. Clin Lab Sci. 2008; 21(4): 199-204.
Schenker JD, Goldstein S, Braun J, et al.: Triage accuracy at a multiple casualty incident disaster drill: The Emergency Medical Service, Fire Department of New York City experience. J Burn Care Res. 2006; 27(5): 570-575.
Wallace DJ, Arquilla B, Heffernan R, et al.: A test of syndromic surveillance using a severe acute respiratory syndrome model. Am J Emerg Med. 2009; 27(4): 419-423.
Claudius I, Behar S, Ballow S, et al.: Disaster drill exercise documentation and management: Are we drilling to standard? J Emerg Nurs. 2008; 34(6): 504-508.
Jones JR, Neff LJ, Ely EK, et al.: Results of medical countermeasure drills among 72 cities readiness initiative metropolitan statistical areas, 2008-2009. Disaster Med Public Health Prep. 2012; 6(4): 357-362.
Kohlhoff SA, Crouch B, Roblin PM, et al.: Evaluation of hospital mass screening and infection control practices in a pandemic influenza full-scale exercise. Disaster Med Public Health Prep. 2012; 6(4): 378-384.
Leiba A, Goldberg A, Hourvitz A, et al.: Who should worry for the “worried well”? Analysis of mild casualties center drills in nonconventional scenarios. Prehosp Disaster Med. 2006; 21(6): 441-444.
Leiba A, Drayman N, Amsalem Y, et al.: Establishing a high level of knowledge regarding bioterrorist threats in Emergency Department physicians: Methodology and the results of a National Bio-Preparedness Project. Prehosp Disaster Med. 2007; 22(3): 207-211; discussion 212-203.
Mackenzie C, Donohue J, Wasylina P, et al.: How will military/civilian coordination work for reception of mass casualties from overseas? Prehosp Disaster Med. 2009; 24(5): 380-388.
Manion P, Golden IJ: Vertical evacuation drill of an intensive care unit: Design, implementation, and evaluation. Disaster Manag Response. 2004; 2(1): 14-19.
Emery RJ, Sprau DD, Morecook RC: Risk communication considerations to facilitate the screening of mass populations for potential contamination with radioactive material. Health Phys. 2008; 95(5 suppl 5): s168-s174.
Adini B, Goldberg A, Cohen R, et al.: Relationship between standards of procedures for pandemic flu and level of hospital performance in simulated drills. Ann Emerg Med. 2008; 52(3): 223-229.
Cone DC, Serra J, Burns K, et al.: Pilot test of the SALT mass casualty triage system. Prehosp Emerg Care. 2009; 13(4): 536-540.
FitzGerald DJ, Sztajnkrycer MD, Crocco TJ: Chemical weapon functional exercise—Cincinnati: Observations and lessons learned from a “typical medium-sized” city's response to simulated terrorism utilizing weapons of mass destruction. Public Health Rep. 2003; 118(3): 205-214.
Caum J, Alles S: Ready or not: Analysis of a no-notice mass vaccination field response in Philadelphia. Biosecur Bioterror. 2013; 11(4): 262-270.
Klein KR, Atas JG, Collins J: Testing emergency medical personnel response to patients with suspected infectious disease. Prehosp Disaster Med. 2004; 19(3): 256-265.
Phin NF, Rylands AJ, Allan J, et al.: Personal protective equipment in an influenza pandemic: A UK simulation exercise. J Hosp Infect. 2009; 71(1): 15-21.
Allen K, Lorek E, Mensia-Joseph N: Conducting a school-based mock drill: Lessons learned from one community. Biosecur Bioterror. 2008; 6(2): 191-201.
Fertel BS, Kohlhoff SA, Roblin PM, et al.: Lessons from the “Clean Baby 2007” pediatric decontamination drill. Am J Disaster Med. 2009; 4(2): 77-85.
Franc-Law JM, Bullard M, Della Corte F: Simulation of a hospital disaster plan: A virtual, live exercise. Prehosp Disaster Med. 2008; 23(4): 346-353.
Freimuth VS, Hilyard KM, Barge JK, et al.: Action, not talk: A simulation of risk communication during the first hours of a pandemic. Health Promot Pract. 2008; 9(4 suppl): 35S-44S.
Beaton RD, Oberle MW, Wicklund J, et al.: Evaluation of the Washington State National Pharmaceutical Stockpile dispensing exercise: Part I—Patient volunteer findings. J Public Health Manag Pract. 2003; 9(5): 368-376.
Stergachis A, Wetmore CM, Pennylegion M, et al.: Evaluation of a mass dispensing exercise in a cities readiness initiative setting. Am J Health Syst Pharm. 2007; 64(3): 285-293.
Beaton RD, Stevermer A, Wicklund J, et al.: Evaluation of the Washington State National Pharmaceutical Stockpile dispensing exercise, part II—Dispensary site worker findings. J Public Health Manag Pract. 2004; 10(1): 77-85.
Craig AT, Armstrong PK: Exercise Paton: A simulation exercise to test New South Wales emergency departments’ response to pandemic influenza. Commun Dis Intell. 2007; 31(3): 310-313.
Lau PF, Lau CC: A disaster drill in Hong Kong. Accid Emerg Nurs. 1997; 5(1): 34-38.
Lifshitz I, Adler J, Katz J: Model for preparedness of a public institution for a terrorist attack. Prehosp Disaster Med. 2008; 23(5): 467-471.
Rifat R: Disaster drill at a university hospital. J Pak Med Assoc. 2005; 55(1): 28-32.
Simpson DM: Earthquake drills and simulations in community-based training and preparedness programmes. Disasters. 2002; 26(1): 55-69.
Emery RJ, Sprau DD, Morecook RC, et al.: Surge capacity volunteer perspectives on a field training exercise specifically designed to emphasize likely roles during a disaster response. Health Phys. 2009; 97(5 suppl): S155-S160.
Lynch TF, Kugler L, Niedziela J: Saving our smallest patients. J Healthc Prot Manage. 2014; 30(2): 72-82.
Ramirez M, Kubicek K, Peek-Asa C, et al.: Accountability and assessment of emergency drill performance at schools. Fam Commun Health. 2009; 32(2): 105-114.
Taylor K, Balfanz-Vertiz K, Humrickhouse R, et al.: Decontamination with at-risk populations: Lessons learned. Internet J Rescue Disaster Med. 2009; 9(1): 39.
Peterson DM, Perry RW: The impact of disaster exercise on participants. Disaster Prev Manag. 1999; 8(4): 241-254.
Kotora JG, Clancy T, Manzon L, et al.: Active shooter in the emergency department: A scenario-based training approach for healthcare workers. Am J Disaster Med. 2014; 9(1): 39-51.
Perry RW: Disaster exercise outcomes for professional emergency personnel and citizen volunteers. J Conting Crisis Manag. 2004; 12(2): 64-75.
Djalali A, Castren M, Hosseinijenab V, et al.: Hospital Incident Command System (HICS) performance in Iran; decision making during disasters. Scand J Trauma, Resusc Emerg Med. 2012; 20: 14.
Mobula LM, Jacquet GA, Weinhauer K, et al.: Community health facility preparedness for a cholera surge in Haiti. Am J Disaster Med. 2013; 8(4): 235-241.
Agboola F, McCarthy T, Biddinger PD: Impact of emergency preparedness exercise on performance. J Public Health Manag Pract. 2013; 19: S77-S83.
Taylor JL, Roup BJ, Blythe D, et al.: Pandemic influenza preparedness in Maryland: Improving readiness through a tabletop exercise. Biosecur Bioterror. 2005; 3(1): 61-69.
Beaton R, Stergachis A, Thompson J, et al.: Pandemic policy and planning considerations for universities: Findings from a tabletop exercise. Biosecur Bioterror. 2007; 5(4): 327-334.
Dausey DJ, Buehler JW, Lurie N: Designing and conducting tabletop exercises to assess public health preparedness for manmade and naturally occurring biological threats. BMC Public Health. 2007; 7: 92.
Doxtator LA, Gardner CE, Medves JM: Responding to pandemic influenza: A local perspective. Can J Public Health. 2004; 95(1): 27-31.
Klein KR, Brandenburg DC, Atas JG, et al.: The use of trained observers as an evaluation tool for a multi-hospital bioterrorism exercise. Prehosp Disaster Med. 2005; 20(3): 159-163.
High EH, Lovelace KA, Gansneder BM, et al.: Promoting community preparedness: Lessons learned from the implementation of a chemical disaster tabletop exercise. Health Promot Pract. 2010; 11(3): 310-319.
Jarrett D: Lessons learned: The “pale horse” bioterrorism response exercise. Disaster Manag Response. 2003; 1(4): 114-118.
Morris JG Jr, Greenspan A, Howell K, et al.: Southeastern center for emerging biologic threats tabletop exercise: Foodborne toxoplasmosis outbreak on college campuses. Biosecur Bioterror. 2012; 10(1): 89-97.
Edwards JC, Kang JE, Silenas R: Promoting regional disaster preparedness among rural hospitals. J Rural Health. 2008; 24(3): 321-325.
Macario E, Heyden L, Nakahara N, et al.: Preparing for pandemic influenza: California confronts the legal implications. Health Promot Pract. 2009; 10(4): 573-578.
Lurie N, Dausey DJ, Knighton T, et al.: Community planning for pandemic influenza: Lessons from the VA health care system. Disaster Med Public Health Prep. 2008; 2(4): 251-257.
Macario E, Benton LD, Yuen J, et al.: Preparing public health nurses for pandemic influenza through distance learning. Public Health Nurs. 2007; 24(1): 66-72.
Atack L, Bull E, Dryden T, et al.: An evaluation of learner perception of competency and satisfaction with three models of an interdisciplinary surge capacity course. J Allied Health. 2012; 41(3): 106-112.
Savoia E, Biddinger PD, Fox P, et al.: Impact of tabletop exercises on participants’ knowledge of and confidence in legal authorities for infectious disease emergencies. Disaster Med Public Health Prep. 2009; 3(2): 104-110.
Henning KJ, Brennan PJ, Hoegg C, et al.: Health system preparedness for bioterrorism: Bringing the tabletop to the hospital. Infect Control Hosp Epidemiol. 2004; 25(2): 146-155.
Bartley B, Fisher J, Stella J: Video of a disaster drill is effective in educating registrars on the hospital disaster plan. Emerg Med Australas. 2007; 19(1): 39-44.
Sarpy SA, Warren CR, Kaplan S, et al.: Simulating public health response to a severe acute respiratory syndrome (SARS) event: A comprehensive and systematic approach to designing, implementing, and evaluating a tabletop exercise. J Public Health Manag Pract. 2005; (suppl): 375-382.
Shah VS, Pierce LC, Roblin P, et al.: Waterworks, a full-scale chemical exposure exercise: Interrogating pediatric critical care surge capacity in an inner-city tertiary care medical center. Prehosp Disaster Med. 2014; 29(1): 100-106.
Lord EJ, Cieslak TJ: Lessons learned. Joint Regional Exercise (“JREX”) 2000. Disaster Manag Response. 2004; 2(1): 24-27.
Klima DA, Seiler SH, Peterson JB, et al.: Full-scale regional exercises: Closing the gaps in disaster preparedness. J Trauma Acute Care Surg. 2012; 73(3): 592-597; discussion 597-598.
Savoia E, Testa MA, Biddinger PD, et al.: Assessing public health capabilities during emergency preparedness tabletop exercises: Reliability and validity of a measurement tool. Public Health Rep. 2009; 124(1): 138-148.
Leiba A, Ashkenasi I, Nakash G, et al.: Response of Thai hospitals to the Tsunami disaster. Prehosp Disaster Med. 2006; 21(1): S32-S37.
Pennell PA, Brennan LE, Burhans RL, et al.: Packaging and shipping capabilities of New York State hospital laboratories: A 3-year drill assessment. Clin Lab Sci. 2011; 24(1): 14-21.
Biddinger PD, Cadigan RO, Auerbach BS, et al.: Using exercises to identify systems-level preparedness challenges. Public Health Rep. 2008; 123(1): 96-101.
Savoia E, Biddinger PD, Burstein J, et al.: Inter-agency communication and operations capabilities during a hospital functional exercise: Reliability and validity of a measurement tool. Prehosp Disaster Med. 2010; 25(1): 52-58.
Ford JK, Schmidt AM: Emergency response training: Strategies for enhancing real-world performance. J Hazard Mater. 2000; 75(2-3): 195-215.
Kirkpatrick D: Evaluatinig Training Programs: The Four Levels. New York: Berrett-Koehler, 1998.
Brooks J, Brooks M: In Search of Understanding: The Case for Constructivist Classrooms. Alexandria, VA: Association for Supervision & Curriculum Development, 1993.
Fanning R, Gaba D: The role of debriefing in simulation-based learning. Simul Healthc. 2007; 2(2): 115-125.
Johnstone AH, Sleet RJ, Vianna JF: An information processing model of learning: Its application to an undergraduate laboratory course in chemistry. Stud High Educ. 1994; 19(1): 77-87.
Savoia E, Agboola F, Biddinger PD: Use of after action reports (AARs) to promote organizational and systems learning in emergency preparedness. Int J Environ Res Public Health. 2012; 9(8): 2949-2963.
Published
How to Cite
Issue
Section
License
Copyright 2007-2023, Weston Medical Publishing, LLC
All Rights Reserved
