Using evaluation theory to augment the homeland security exercise and evaluation program (HSEEP) guidance for evaluating operations-based exercises

Authors

  • Ralph Renger, PhD, MEP
  • Anneke Jansen, MPH
  • Erin Peacock, MPH
  • Adriana Cimetta, MPH
  • Jessica Surdam, MPH

DOI:

https://doi.org/10.5055/jem.2008.0021

Keywords:

evaluating, HSEEP, operations-based, exercises, theory

Abstract

Exercises play a crucial role in better preparing for, responding to, and recovering from an emergency by providing opportunities for responders and officials to practice and assess their collective capabilities. Conducting a thorough evaluation of these exercises is critical to ensuring that the nation continually improves its ability to save lives and property. A major emphasis of the Homeland Security Exercise and Evaluation Program (HSEEP) is on defining and evaluating capability- based objectives to determine the impact of an exercise. Using the integrated theory of evaluation, it is shown how a cost-effective, quality evaluation of operations- based exercises can be conducted while simultaneously not interfering or adding to the burden of exercise players, controllers, or evaluators. It is hoped that this article will act as a catalyst in moving HSEEP to recognize the potential of other sources of information to assist in conducting a more comprehensive evaluation and amend their guidelines accordingly.

Author Biographies

Ralph Renger, PhD, MEP

University of Arizona, College of Public Health, and Arizona Center for Public Health Preparedness, Tucson, Arizona.

Anneke Jansen, MPH

University of Arizona, College of Public Health, and Arizona Center for Public Health Preparedness, Tucson, Arizona.

Erin Peacock, MPH

University of Arizona, College of Public Health, Tucson, Arizona.

Adriana Cimetta, MPH

University of Arizona, College of Public Health, Tucson, Arizona.

Jessica Surdam, MPH

University of Arizona, College of Public Health, Tucson, Arizona.

References

United States Office of Management and Budget [OMB]: Detailed information on the Federal Emergency Management Agency: Grants and Training Office National Exercise Program Assessment [Web page]. Available at http://www.whitehouse.gov/omb/expectmore/detail/10003606.2005.html. Accessed October 16, 2007.

Redlener I: Americans at Risk: Why We Are Not Prepared for Megadisasters and What We Can Do. New York, New York: Knopf Publishing Group, 2006.

Federal Emergency Management Agency: ICS-100: Introduction to ICS Student Manual. Emmitsburg, MD: Emergency Management Institute, 2005.

United States Department of Homeland Security: Homeland Security and Evaluation Program (HSEEP), Volume 1: HSEEP overview and exercise program management. Available at https://hseep.dhs.gov/support/VolumeI.pdf. Accessed May 1, 2007.

United States Department of Homeland Security: Homeland Security Exercise and Evaluation Program: Volume II exercise planning and conduct. Available at: https://hseep.dhs.gov/support/ VolumeII.pdf. Accessed May 1, 2007.

United States Department of Homeland Security: Homeland Security Exercise and Evaluation Program: Volume III exercise evaluation and improvement planning. Available at https://hseep.dhs.gov/support/VolumeIII.pdf. Accessed May 1, 2007.

Department of Homeland Security: Homeland Security Exercise and Evaluation Program, terminology, methodology, and compliance guidelines.Available at https://hseep.dhs.gov/support/HSEEP_101.pdf. Accessed February 7, 2008.

Federal Emergency Management Agency: IS-130: Exercise Evaluation and Improvement Planning. Emmitsburg, MD: Emergency Management Institute, 2008.

Federal Emergency Management Agency: Homeland security exercise and evaluation program terminology, methodology and compliance guidelines. Available at https://hseep.dhs.gov/support/HSEEP%20101.pdf. Accessed October 3, 2007.

Federal Emergency Management Agency: ICS-700: National Incident Management System (NIMS), An Introduction Self-Study Guide. Emmitsburg, MD: Emergency Management Institute, 2004.

United States Department of Homeland Security: National Incident Management System. Available at http://www.fema.gov/pdf/emergency/nims/nims_doc_full.pdf. Accessed May 1, 2007.

United State Department of Homeland Security: Target capabilities list a companion to the national preparedness guideline, 2007. Available at https://www.llis.dhs.gov/getFile.cfm?id=26724. Accessed October 16, 2007.

Mark MM, Henry GT, Julnes G. Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco: Jossey Bass, 2000.

Office of Management and Budget: Government Performance Results Acts (GPRA). Washington, DC: The White House, 1993. Available at http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html. Accessed May 1, 2007.

Conrad E, Maul T. Introduction to Experimental Psychology. New York, NY:Wiley, 1981.

Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standards: How to Assess Evaluations of Educational Programs, 2nd ed. Thousand Oaks, CA: Sage, 1994.

Renger R Consequences to Federal Programs when the logic modeling process is not followed with fidelity. Am J Eval. 2006; 27(4): 452-464.

Federal Emergency Management Agency. ICS Resource Center: ICS Forms [Web page]. Available at http://training.fema.gov/emiweb/icsresource/ICSResCntr_Forms.htm. Accessed May 1, 2007.

Published

05/01/2008

How to Cite

Renger, PhD, MEP, R., A. Jansen, MPH, E. Peacock, MPH, A. Cimetta, MPH, and J. Surdam, MPH. “Using Evaluation Theory to Augment the Homeland Security Exercise and Evaluation Program (HSEEP) Guidance for Evaluating Operations-Based Exercises”. Journal of Emergency Management, vol. 6, no. 3, May 2008, pp. 45-52, doi:10.5055/jem.2008.0021.

Issue

Section

Articles