The SERIES model: Can a standardized approach benefit practitioner evaluation of emergency response systems?
DOI:
https://doi.org/10.5055/jem.0582Keywords:
emergency response, system evaluation, organizational learning, network analysis, practitioner, fire serviceAbstract
The purpose of this research was to investigate the use of a standardized model of data collection, storage, and interpretation to evaluate the organized response to emergencies. The Standardized Emergency Response Incident Evaluation System (SERIES) model was employed to support practitioner evaluation of a fire service response to a simulated high-rise building fire. A comprehensive dataset representing quantitative and qualitative characteristics of the emergency response is established through content analysis of documents produced during the response and interviews with responders post-incident. Data are encoded within a standardized framework employing network theory. This dataset is then evaluated by fire service subject matter experts to identify learning outcomes relevant to emergency response effectiveness and organizational development. The results demonstrate that emergency response management system data can be collected and stored within a standardized framework. The results also demonstrate that not only can the resultant dataset support the identification of a range of learning outcomes by fire service practitioners, but that this approach can lead to a more comprehensive understanding of the response and deeper learning than that provided by existing evaluation methods.
References
Bergstrom J, Uhr C, Frykmer T: A complexity framework for studying disaster response management. J Contingencies Cris Manag. 2016; 24(3): 124-135.
Groenendaal J, Helsloot I: A closer examination of command and control practices by incident commanders during realistic operational exercises in the Netherlands. Int J Emerg Manag. 2018; 14(1): 51-71.
Steigenberger N: Organizing for the big one: A review of case studies and a research agenda for multi-agency disaster response. J Contingencies Cris Manag. 2016; 24(2): 60-72.
Beerens RJJ: Does the means achieve an end? A document analysis providing an overview of emergency and crisis management evaluation practice in the Netherlands. Int J Emerg Manag. 2019; 15(3): 221-254.
Bowen AA: Are we really ready? The need for national emergency preparedness standards and the creation of the cycle of emergency planning. Polit Policy. 2008; 36(5): 834-853.
Pilemalm S, Andersson D, Hallber N: Reconstruction and exploration of large-scale distributed operations—Multimedia tools for evaluation of emergency management response. J Emerg Manag. 2008; 6(6): 31-47.
Comfort LK: Inter-Organizational Design for Disaster Management: Cognition, communication, coordination, and control. J Seismol Earthq Eng. 2007; 9(1,2): 61-71.
Abrahamsson M, Hassel H, Tehler H: Towards a system-oriented framework for analysing and evaluating emergency response. J Contingencies Cris Manag. 2010; 18(1): 14-25.
Pollock K: Review of persistent lessons identified relating to interoperability from emergencies and major incidents since 1986. Emergency Planning College, Occasional Papers New Series, No. 6; 2013.
Coles E: Learning the lessons from major incidents: A short review of the literature. Emergency Planning College, Occasional Papers Series, No.10; 2014. Available at https://www.epcresilience.com/EPC.Web/media/documents/Papers/Occ10-Paper.pdf. Accessed October 10, 2018.
Cabinet Office: Emergency Response and Recovery: Non Statutory Guidance Accompanying the Civil Contingencies Act 2004; 2013.
NHS England, NHS Improvement: National guidance for ambulance trusts on learning from deaths: A framework for NHS ambulance trusts in England on identifying, reporting, reviewing and learning from deaths in care; 2019. Available at https://www.england.nhs.uk/wp-content/uploads/2019/07/learning-from-deathsguidance-for-ambulance-trusts.pdf. Accessed July 10, 2020.
National Fire Chiefs Council: National operational learning: Good practice guide for fire and rescue services; 2019. Available at https://www.ukfrs.com/sites/default/files/2017-09/GoodPracticeGuideforNationalOperationalLearning.pdf. Accessed July 10, 2020.
Holdsworth D, Zagorecki A: The SERIES Model: Development of a practitioner focused emergency response evaluation system. Int J Emerg Serv. 2020; 9(3): 313-337.
Farjoun M, Ansell C, Boin A: Perspective - Pragmatism in organization studies: Meeting the challenges of a dynamic and complex world. Organ Sci. 2015; 26(6): 1787-1804.
Whitford J, Zirpoli F: Pragmatism, practice, and the boundaries of organization. Organ Sci. 2014; 25(6): 1823-1839.
Corbacioglu S, Kapucu N: Organisational learning and self-adaptation in dynamic disaster environments. Disasters 2006; 30(2): 212-233.
Kapucu N: Interorganizational coordination in dynamic context: Networks in emergency response. Connections. 2005; 26(2): 33-48.
Chen R, Sharman R, Rao HR, et al.: Coordination in emergency response management. Commun ACM. 2008; 51(5): 66-73.
Eide AW, Haugstveit IM, Halvorsrud R: Key challenges in multiagency collaboration during large-scale emergency management. In Divitini M, Farshchian BA, Floch J, et al. (eds.): Proceedings of the Workshop on Ambient Intelligence for Crisis Management, A Workshop in Conjunction with the International Joint Conference on Ambient Intelligence (AMI2012), Pisa, November 13, 2012, CEUR-WS.org, Aachen. Available at http://ceur-ws.org/Vol-953/. Accessed April 5, 2020.
Bharosa N, Lee J, Janssen M: Challenges and obstacles in sharing and coordinating information during multi-agency disaster response: Propositions from field exercises. Inf Syst Front. 2010; 12: 49-65.
Comfort LK: Crisis management in hindsight: Cognition, communication, coordination, and control. Public Adm Rev. 2007; 67(s1): 189-198.
Khandekar A, Sharma A: Organizational learning and performance: Understanding Indian scenario in present global context. Educ Train. 2006; 48(8-9): 682-692.
Odor HO: A literature review on organizational learning and learning organizations. Int J Econ Manag Sci. 2018; 7(1).
Senge PM: Taking personal change seriously: The impact of Organizational Learning on management practice. Acad Manag Exec. 2003; 17(2): 47-50.
Carroll JS, Rudolph JW, Hatakenaka S: Learning from experience in high-hazard organizations. Res Organ Behav. 2002; 24: 87-137.
March JG, Sproull LS, Tamuz M: Learning from samples of one or fewer. Organ Sci. 1991; 2(1): 1-13.
Chiva R, Alegre J: Organizational learning and organizational knowledge towards the integration of two approaches. Manag Learn. 2005; 36(1): 49-68.
Dorasamy M, Jennex ME, Raman M, et al.: Knowledge management for disaster response: A proposed framework. In Hashim NL, Shiratuddin N, Baharom F, et al. (eds.): Proceedings of Knowledge Management International Conference 2008 (KMICE). Langkawi, Malaysia: Universiti Utara Malaysia, 2008: 257-263.
Bouthillier F, Shearer K: Understanding knowledge management and information management: The need for an empirical perspective. Inf Res. 2002; 8(1).
Ashritha KS, Prajwala TM, Chandrasekaran K: Activity theory based approach for requirements analysis of Android Applications. In Uden L, Lu W, Ting I (eds.): Knowledge management in organizations. 12th international conference (kmo 2017). Communications in computer and information science (Vol. 731). Beijing, China: Springer, Cham, 2017: 3-15.
Škerlavaj M, Dimovski V: Towards network perspective of intraorganizational learning: Bridging the gap between acquisition and participation perspective. Interdiscip J Information, Knowledge, Manag. 2007; 2: 43-58.
Easterby-Smith M, Crossan M, Davide N: Organizational learning: Debates past, present and future. J Manag Stud. 2000; 37(6): 783-796.
Fiol CM, Lyles MA: Organizational learning. Acad Manag Rev. 1985; 10(4): 803-813.
Argyris C, Schon DA: Organizational Learning: A Theory of Action Perspective. Reading, MA: Addison-Wesley Publishing Company, 1978.
March JG: Exploration and exploitation in organizational learning. Organ Sci. 1991; 2(1): 71-88.
Meyer AD: Adapting to environmental jolts. Adm Sci Q. 1982; 27(4): 515-537.
Glassey S: Preventing ‘lessons lost’: Is evidence-based dynamic doctrine the answer? Aust J Emerg Manag. 2015; 30(3): 11-14.
Moynihan DP: From intercrisis to intracrisis learning. J Contingencies Cris Manag. 2009; 17(3): 189-198.
Pilemalm S, Andersson D, Mojir KY: Enabling organizational learning from rescue operations: The Swedish rescue services incident reporting system. Int J Emerg Serv. 2014; 3(2): 101-117.
Bruce D: Post-incident research - Gaining knowledge after the event. Aust J Emerg Manag. 2015; 30(3): 6-8.
Zhou S, Battaglia M, Frey M: Organizational learning through disasters: A multi-utility company’s experience. Disaster Prev Manag An Int J. 2018; 27(2): 243-254.
Carley KM, Harrald JR: Organizational Learning under fire: Theory and practice. Am Behav Sci. 1997; 40(3): 310-332.
Metallinou MM: Single- and double-loop organizational learning through a series of pipeline emergency exercises. J Contingencies Cris Manag. 2018; 26(4): 530-543.
Thompson DDP: Leveraging learning to improve disaster management outcomes. Int J Disaster Risk Sci. 2012; 3(4): 195-206.
Heath R: Looking for answers: Suggestions for improving how we evaluate crisis management. Saf Sci. 1998; 30(1-2): 151-163.
Scriven M: The methodology of evaluation. In Tyler R, Gagne R, Scriven M (eds.): Perspectives on Curriculum Evaluation (AERA Monograph Series – Curriculum Evaluation). Chicago: Rand McNally and Co, 1967.
Jillson IA, Clarke M, Allen C, et al.: Improving the science and evidence base of disaster response: A policy research study. BMC Health Serv Res. 2019; 19(274).
Pearce W, Raman S: The new randomised controlled trials (RCT) movement in public policy: Challenges of epistemic governance. Policy Sci. 2014; 47(4): 387-402.
Hertting N, Vedung E: Purposes and criteria in network governance evaluation: How far does standard evaluation vocabulary takes us? Evaluation 2012; 18(1): 27-46.
Beerens RJJ, Tehler H, Pelzer B: How can we make disaster management evaluations more useful? An empirical study of Dutch exercise evaluations. Int J Disaster Risk Sci. 2020; 11: 578-591.
Gillespie D, Streeter C: Conceptualizing and measuring disaster preparedness. Int J Mass Emerg Disasters 1987; 5(2): 155-176.
Henstra D: Evaluating local government emergency management programs: What framework should public managers adopt? Public Adm Rev. 2010; 70(2): 236-246.
Morin M, Jenvald J, Thorstensson M: Computer-supported visualization of rescue operations. Saf Sci. 2000; 35(1-3): 3-27.
Ellis S, Mendel R, Nir M: Learning from successful and failed experience: The moderating role of kind of after-event review. J Appl Psychol. 2006; 91(3): 669-680.
Ellis S, Davidi I: After-event reviews: Drawing lessons from successful and failed experience. J Appl Psychol. 2005; 90(5): 857-871.
Kapucu N, Yuldashev F, Feldheim MA: Nonprofit organizations in disaster response and management: A network analysis. Eur J Econ Polit Stud. 2011; 4(1): 83-112.
Hu Q, Knox CC, Kapucu N: What have we learned since September 11, 2001? A network study of the boston marathon bombings response. Public Adm Rev. 2014; 74(6): 698-712.
Kapucu N, Augustin M-E, Garayev V: Interstate partnerships in emergency management: Emergency management assistance compact in response to catastrophic disasters. Public Adm Rev. 2009; 69(2): 297-313.
Van Der Haar S, Segers M, Jehn KA: Measuring the effectiveness of emergency management teams: Scale development and validation. Int J Emerg Manag. 2013; 9(3): 258-275.
Savoia E, Agboola F, Biddinger PD: A conceptual framework to measure systems’ performance during emergency preparedness exercises. Int J Environ Res Public Health. 2014; 11(9): 9712-9722.
Wybo JL: The role of simulation exercises in the assessment of robustness and resilience of private or public organisations. In Pasman HJ, Kirillov IA (eds.): Resilience of cities to terrorist and other threats: Learning from 9/11 and further research issues. Dordrecht: Springer, 2008: 491–507.
Sinclair H, Doyle EE, Johnston DM, et al.: Assessing emergency management training and exercises. Disaster Prev Manag Int J. 2012; 21(4): 507-521.
Da Costa Duarte A, Da Silva Borges MR, Gomes JO, et al.: ASC model: A process model for the evaluation of simulated field exercises in the emergency domain. In Comes T, Fiedrich F, Fortier S, et al. (eds.): 10th International Conference on Information Systems for Crisis Response and Management. Ocean City, MA: ISCRAM Association, 2013: 551-555.
Heumüller E, Richter S, Lechner U: Towards a framework for command post exercises. In ISCRAM 2012 Conference Proceedings - 9th International Conference on Information Systems for Crisis Response and Management. Vancouver, Canada: Simon Fraser University, 2012.
Glassey S: Preventing lessons lost through an evidence based dynamic doctrine. Working Paper, Canterbury Institute for Risk, Resilience and Renewal, University of Canterbury, 2014. Available at http://hdl.handle.net/10092/10066. Accessed October 13, 2020.
Kapucu N, Garayev V: Structure and network performance: Horizontal and vertical networks in emergency management. Adm Soc. 2016; 48(8): 931-961.
Abbasi A, Hossain L, Hamra J, et al.: Social networks perspective of firefighters’ adaptive behaviour and coordination among them. In Zhu P, Wang L, Xia F, et al. (eds.): IEEE/ACM International Conference on Green Computing and Communications and International Conference on Cyber, Physical and Social Computing. Hangzhou, China, 2010: 819-824.
Mandell M, Keast R: Evaluating network arrangements: Toward revised performance measures. Public Perform Manag Rev. 2007; 30(4): 574-597.
Hamra J, Hossain L, Owen C, et al.: Effects of networks on learning during emergency events. Disaster Prev Manag. 2012; 21(5): 584-598.
Comfort LK, Dunn M, Johnson DEA, et al.: Coordination in complex systems: Increasing efficiency in disaster mitigation and response. Int J Emerg Manag. 2004; 2(1-2): 62-80.
Uhr C, Johansson H: Mapping an emergency management network. Int J Emerg Manag. 2007; 4(1): 104-118.
Noori NS, Paetzold K, Vilasis-Cardona X: Network based discrete event analysis for coordination processes in crisis response operations. In Annual IEEE Systems Conference (SysCon). Orlando, FL: IEEE, 2016: 596-600.
Abbasi A: Link formation pattern during emergency response network dynamics. Nat Hazards. 2014; 71(3): 1957-1969.
Andrew T: Analysis of command and control networks on Black Saturday. Aust J Emerg Manag. 2011; 26(3): 20-29.
Kapucu N, Garayev V: Designing, managing, and sustaining functionally collaborative emergency management networks. Am Rev Public Adm. 2012; 43(3): 312-330.
Liu D, Wang H, Qi C, et al.: Hierarchical task network-based emergency task planning with incomplete information, concurrency and uncertain duration. Knowledge-Based Syst. 2016; 112: 67-79.
Zhao P, Qi C, Liu D: Resource-constrained hierarchical task network planning under uncontrollable durations for emergency decision-making. J Intell Fuzzy Syst. 2017; 33(6): 3819-3834.
Dahlberg R: Who is in the center? A case study of a social network in an emergency management organization. Int J Emerg Serv. 2017; 6(1): 52-66.
Kapucu N, Demiroz F: Measuring performance for collaborative public management using network analysis methods and tools. Public Perform Manag Rev. 2011; 34(4): 549-579.
Hanneman RA, Riddle M: Concepts and measures for basic network analysis. In Scott J, Carrington PJ (eds.): The Sage Handbook of Social Network Analysis. London: SAGE Publications Ltd, 2016: 340-369.
Hossain L, Kuti M: Disaster response preparedness coordination through social networks. Disasters. 2010; 34(3): 755-786.
Zhao J, Wang Y, Yu L: Applying process mining techniques to improve emergency response planning for chemical spills. Int J Disaster Risk Reduct. 2019; 34(March): 184-195.
Clark-Ginsberg A, Abolhassani L, Rahmati EA: Comparing networked and linear risk assessments: From theory to evidence. Int J Disaster Risk Reduct. 2018; 30(part B): 216-224.
Kapucu N: Interagency communication networks during emergencies: Boundary spanners in multiagency coordination. Am Rev Public Adm. 2006; 36(2): 207-225.
Klimek P, Varga J, Jovanovic AS, et al.: Quantitative resilience assessment in emergency response reveals how organizations trade efficiency for redundancy. Saf Sci. 2019; 113: 404-414.
Batool K, Niazi MA: Towards a methodology for validation of centrality measures in complex networks. PLoS One 2014; 9(4): e98379.
Valente TW, Coronges K, Lakon C, et al.: How correlated are network centrality measures? Connect 2008; 28(1): 16-26.
Moore S, Daniel M, Eng E: International NGOs and the role of network centrality in humanitarian aid operations: A case study of coordination during the 2000 Mozambique floods. Disasters 2003; 27(4): 305-318.
Ongkowijoyo CS, Doloi H, Gurmu AT: Hybrid risk analysis model for analyzing the urban infrastructure risk. Int J Disaster Risk Reduct. 2020; 48.
Provan KG, Lemaire RH: Core concepts and key ideas for understanding public sector organizational networks: Using research to inform scholarship and practice. Public Adm Rev. 2012; 72(5): 638-648.
Dunn JC, Lewandowsky S, Kirsner K: Dynamics of communication in emergency management. Appl Cogn Psychol. 2002; 16(6): 719-737.
Alison L, Power N, van den Heuvel C, et al.: Decision inertia: Deciding between least worst outcomes in emergency responses to disasters. J Occup Organ Psychol. 2015; 88(2): 295-321.
Grunwald JA, Bearman C: Identifying and resolving coordinated decision making breakdowns in emergency management. Int J Emerg Manag. 2017; 13(1): 68-89.
Ryan B: The significance of communication in emergency management: What’s changed since 2010? Aust J Emerg Manag. 2017; 32(1): 24-31.
Alison L, van den Heuvel C, Waring S, et al.: Immersive simulated learning environments for researching critical incidents: A knowledge synthesis of the literature and experiences of studying high-risk strategic decision making. J Cogn Eng Decis Mak. 2013; 7(3): 255-272.
Cohen-Hatton SR, Honey RC: Goal-oriented training affects decision-making processes in virtual and simulated fire and rescue environments. J Exp Psychol Appl. 2015; 21(4): 395-406.
Berlin JM, Carlström ED: The three-level collaboration exercise - Impact of learning and usefulness. J Contingencies Cris Manag. 2015; 23(4): 257-265.
Berlin JM, Carlström ED: The dominance of mechanistic behaviour: A critical study of emergency exercises. Int J Emerg Manag. 2013; 9(4): 327-350.
Kristiansen E, Løwe Sørensen J, Carlström E, et al.: Time to rethink Norwegian maritime collaboration exercises. Int J Emerg Serv. 2017; 6(1): 14-28.
Edzen S, Sein M: Designing theme-based tabletop exercise for identifying and dealing with coordination problems in emergencies. Int J Emerg Manag. 2016; 12(1): 22-40.
Skryabina EA, Betts N, Reedy G, et al.: The role of emergency preparedness exercises in the response to a mass casualty terrorist incident: A mixed methods study. Int J Disaster Risk Reduct. 2020; 46.
Beerens RJJ, Tehler H: Scoping the field of disaster exercise evaluation - A literature overview and analysis. Int J Disaster Risk Reduct. 2016; 19: 413-446.
Witt E, Lill I: Methodologies of contemporary disaster resilience research. Procedia Eng. 2018; 212: 970-977.
Wolbers J, Boersma K: The common operational picture as collective sensemaking. J Contingencies Cris Manag. 2013; 21(4): 186-199.
Prytz EG, Rybing J, Jonson CO: Workload differences across command levels and emergency response organizations during a major joint training exercise. J Emerg Manag. 2016; 14(4): 289-297.
Campbell JL, Quincy C, Osserman J, et al.: Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociol Methods Res. 2013; 42(3): 294-320.
Dilmaghani RB, Rao RR: A systematic approach to improve communication for emergency response. In Sprague RH (eds.): Proceedings of the 42nd Annual Hawai’i International Conference on System Sciences. Los Alamitos: IEEE Computer Society. 2009.
Hancox G, Hignett S, Pillin H, et al.: Systems mapping for technology development in CBRN response. Int J Emerg Serv. 2018; 7(2): 111-119.
Danielsson E, Alvinius A, Larsson G: From common operating picture to situational awareness. Int J Emerg Manag. 2014; 10(1): 28-47.
Castillo-Montoya M: Preparing for interview research: The interview protocol refinement framework. Qual Rep. 2016; 21(5): 811-831.
Magnussen LI, Carlstrøm E, Sørensen JL, et al.: Learning and usefulness stemming from collaboration in a maritime crisis management exercise in Northern Norway. Disaster Prev Manag An Int J. 2018; 27(1): 129-140.
Andrew SA, Arlikatti S, Chatterjee V, et al.: Ebola crisis response in the USA: Communication management and SOPs. Int J Disaster Risk Reduct. 2018; 31: 243-250.
Launder D, Perry C: A study identifying factors influencing decision making in dynamic emergencies like urban fire and rescue settings. Int J Emerg Serv. 2014; 3(2): 144-161.
Redshaw S, Ingham V, Loftus S: Emergency decision making: An exploration of tensions between communities of practice. Int J Emerg Manag. 2015; 11(1): 62-75.
Donley RD, Ashcraft MH: The methodology of testing naïve beliefs in the physics classroom. Mem Cognit. 1992; 20(4): 381-391.
Morgan CA, Southwick S: Perspective: I believe what I remember, but it may not be true. Neurobiol Learn Mem. 2014; 112: 101-103.
Hope L, Blocksidge D, Gabbert F, et al.: Memory and the operational witness: Police officer recall of firearms encounters as a function of active response role. Law Hum Behav. 2016; 40(1): 23-35.
Published
How to Cite
Issue
Section
License
Copyright 2007-2023, Weston Medical Publishing, LLC and Journal of Emergency Management. All Rights Reserved
