Workload differences across command levels and emergency response organizations during a major joint training exercise

Erik G. Prytz, PhD; Jonas Rybing, MSc; Carl-Oscar Jonson, PhD

doi:10.5055/jem.2016.0294

Authors

Erik G. Prytz, PhD
Jonas Rybing, MSc
Carl-Oscar Jonson, PhD

DOI:

https://doi.org/10.5055/jem.2016.0294

Keywords:

exercise, live simulation, workload, training, NASA TLX

Abstract

Objective: This study reports on an initial test using a validated workload measurement method, the NASA Task Load Index (TLX), as an indicator of joint emergency exercise effectiveness. Prior research on emergency exercises indicates that exercises must be challenging, ie, result in high workload, to be effective. However, this is often problematic with some participants being underloaded and some overloaded. The NASA TLX was used to test for differences in workload between commanders and subordinates and among three different emergency response organizations during a joint emergency exercise.

Design: Questionnaire-based evaluation with professional emergency responders.

Setting: The study was performed in conjunction with a large-scale interorganizational joint emergency exercise in Sweden.

Participants: A total of 20 participants from the rescue services, 12 from the emergency medical services, and 12 from the police participated in the study (N = 44). Ten participants had a command-level role during the exercise and the remaining 34 were subordinates.

Main Outcome Measure(s): The main outcome measures were the workload subscales of the NASA TLX: mental demands, physical demands, temporal demands, performance, effort, and frustration.

Results: The results showed that the organizations experienced different levels of workload, that the commanders experienced a higher workload than the subordinates, and that two out of three organizations fell below the twenty-fifth percentile of average workload scores compiled from 237 prior studies.

Conclusions: The results support the notion that the NASA TLX could be a useful complementary tool to evaluate exercise designs and outcomes. This should be further explored and verified in additional studies.

Author Biographies

Erik G. Prytz, PhD

Department of Computer and Information Science, Linköping University, Linköping, Sweden.

Jonas Rybing, MSc

Department of Computer and Information Science, Linköping University, Linköping, Sweden.

Carl-Oscar Jonson, PhD

Centre for Teaching and Research in Disaster Medicine and Traumatology, Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden

References

Juffermans J, Bierens J: Recurrent medical response problems during five recent disasters in the Netherlands. Prehosp Disaster Med. 2010; 25(2): 127-136.

Kapucu N: Collaborative emergency management: Better community organising, better public preparedness and response. Disasters. 2008; 32(2): 239-262.

Mishra JL, Allen DK, Pearman AD: Information sharing during multi-agency major incidents. Proc Am Soc Info Sci Tech. 2011; 48(1): 1-10.

Berlin JM, Carlström E: Collaboration exercises: What do they contribute? J Contingencies Crisis Manage. 2015; 23: 11-23.

Berlin JM, Carlström E: Samverkansövningar: Om lärande och nytta. Göteborg, Sweden: Bokförlaget BAS, 2015.

Babus S, Hodges K, Kjonnerod E: Simulations and institutional change: Training US Government professionals for improved management of complex emergencies abroad. J Contingencies Crisis Manage. 1997; 5(4): 231-240.

Quarantelli EL: Disaster crisis management. J Manage Stud. 1988; 25(4): 373-385.

Andersson A, Carlstrom ED, Ahlgren B, et al.: Managing boundaries at the accident scene—A qualitative study of collaboration exercises. Int J Emerg Serv. 2014; 3(1): 77-94.

Lee Y-I, Trim P, Upton J, et al.: Large emergency-response exercises: Qualitative characteristics - A survey. Simul Gaming. 2009; 40(6): 726-751.

Perry RW: Disaster exercise outcomes for professional emergency personnel and citizen volunteers. J Contingencies Crisis Manage. 2004; 12(2): 64-75.

Peterson DM, Perry RW: The impacts of disaster exercises on participants. Disaster Prev Manage. 1999; 8: 241-255.

Moynihan DP: From intercrisis to intracrisis learning. J Contingencies Crisis Manage. 2009; 17(3): 189-198.

Piaget J: The Principles of Genetic Epistemology. New York, NY: Basic Books, 1972.

Van de Ven AH, Poole MS: Explaining development and change. Acad Manage Rev. 1995; 20(3): 510-540.

Hart SG: NASA-Task Load Index (NASA-TLX); 20 years later. Proc Hum Factors Ergonomics Soc Annu Meeting. 2006; 50(9): 904-908.

Hart SG, Staveland LE: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Hancock PA, Meshkati N (eds.): Human Mental Workload. Amsterdam, Netherlands: North Holland Press, 1988: pp. 239-250.

Grier RA: How high is high? A meta-analysis of NASA-TLX global workload scores. Paper presented at 59th Annual Meeting of the Human Factors and Ergonomics Society, Los Angeles, CA, 2015: 1727-1731.

Witmer BG, Singer MJ: Measuring presence in virtual environments: A presence questionnaire. Presence. 1998; 7(3): 225-240.

Csikszentmihalyi M: Flow: The Psychology of Optimal Experience. New York, NY: Harper Perennial, 1990.

Jennett C, Cox AL, Cairns P, et al.: Measuring and defining the experience of immersion in games. Int J Hum Comput Stud. 2008; 66(9): 641-661.

Canossa A, Drachen A, Sørensen JRM: Arrrgghh!!! – Blending quantitative and qualitative methods to detect player frustration. Paper presented at Proceedings of the 6th International Conference on Foundations of Digital Games - FDG ’11. Bordeaux, France. 2011: 61-68.

Feinstein AH, Cannon HM: Fidelity, verifiability, and validity of simulation: Constructs for evaluation, Simul Gaming. 2001; 33(4): 57-67.

Cannon-bowers JA, Salas E, Tannenbaum SI, et al.: Toward theoretically based principles of training effectiveness: A model and initial empirical investigation. Milit Psychol. 1995; 7(3): 141-164.

Perry M: Distributed cognition. In Carroll JM (ed.), IHCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science. San Francisco, CA: Morgan Kauffman Publishers, 2003: 212-223.

Blandford A, Furniss D: DiCoT: A methodology for applying distributed cognition to the design of teamworking systems. In Gilroy S, Harrison M (eds.), 12th International Workshop on Design, Specification and Verification of Interactive Systems. Berlin, Germany: Springer-Verlag, 2006: 26-38.

Rybing J, Nilsson H, Jonson C-O, et al.: Studying distributed cognition of simulation-based team training with DiCoT. Ergonomics. 2015; 59(3): 1-12.

Helton WS, Shaw TH, Warm JS, et al.: Effects of warned and unwarned demand transitions on vigilance performance and stress. Anxiety Stress Coping. 2008; 21(2): 173-184.

Morgan JF, Hancock PA: The effect of prior task loading on mental workload: An example of hysteresis in driving. Hum Factors. 2011; 53(1): 75-86.

Moroney BW, Warm JS, Dember WN: Effects of demand transitions on vigilance performance and perceived workload. Proc Hum Factors Ergonomics Soc Annu Meeting. 1995; 39(21): 1375-1379.

Prytz EG, Scerbo MW: Changes in stress and subjective workload over time following a workload transition. Theor Issues Ergon Sci. 2015; 16(6): 586-605.

Prytz EG, Rybing J, Carlström E, et al.: Exploring prehospital C2-work during a mass gathering event. Int J Emerg Serv. 2015; 4(2): 227-241.