EXERCISE EVALUATION AND FOLLOW-UP CHAPTER 7 EXERCISE EVALUATION AND FOLLOW-UP The evaluation stage is an integral part of the exercise planning cycle and should reflect on the exercise purpose and objectives. Thus, the evaluation work begins already during the planning phase. The exercise team should key in on several questions: What do you want to achieve with the exercise? How will evaluation be done? According to which matrix, methodology, and criteria will the exercise be assessed? If you do not know what you want to measure, assess or compare, it will be difficult to ask the right questions. There are two basic purposes/levels of an evaluation: The first one tells us how to improve the planning process and the exercise itself. It is based on the exercise creators’ observations and participants’ comments about the organization aspects of the exercise. This kind of feedback helps organizers better customize exercises to specific audiences and make exercise documents more comprehendible for them. Based on participant responses, the exercise can be designed so as to cover the issues and topics most relevant to them. Questions asked should focus on areas of improvement and aspects of overall exercise design. This is the right place to ask players about whether they thought the scenario and injects were challenging and realistic, what worked and what did not. Sample evaluation questions:      Was the exercise design realistic? Did the scenario accurately reflect realworld examples? Were the exercise documents and visual materials complete/useful? Were the main events and injects structured and well-organized? Was the introductory presentation helpful in understanding exercise objectives?  Did trainees feel under pressure and stress? The second level of evaluation emphasizes systematic assessment of the exercise objective(s), examination of injects and results, and overall responder performance. Put simply, this type of evaluation focuses on bullet points covered in the “content” part in Figure 7 (previously introduced). The goal is to determine how participants managed to treat the individual tasks and to what level they performed (low-standard, average or superior). More importantly, it brings to light the strengths and weaknesses, gaps, and shortcomings captured when dealing with a complex crisis scenario. Sample evaluation questions:     List improvements that need to be made to plans and procedures for responding to the exercise crisis scenario. List plans, processes or procedures that should be developed, reviewed or revised based on the exercise findings. For each point, indicate who or what agency/unit should be assigned responsibility for implementation. What lessons learned have you identified from the exercise regarding the content aspect? Is there anything in particular you would like to see in future exercises? In order to compile all participant notes and observations, the target audience can be required to fill out specific questionnaires after the end of the exercise (see sample evaluation questions above). In addition, interviews with key players and observers may be helpful. Again, all questions articulated here must be in line with exercise objectives, evaluation metrics and criteria to reveal the greatest challenges and pitfalls. Inputs for evaluation should be gathered during the hot-wash. This is 25/29