Activate Us

Our activation procedure including contact details and forms for activation, in addition to equipment stockpile and aviation status reports can be found here.

Emergency Contact Numbers

Please note that:

Response services are guaranteed ONLY for Members. Non-members are not guaranteed a response and will be required to sign a Non-member contract. Services and rates differ. Duty managers can be contacted for exercises.

Oil Spill Response Limited Oil Spill Response Limited

Welcome to OSRL

  • Article
  • Exercises

Crisis Exercises: Are You Wasting Your Time and Money? A Blueprint for Better Exercises

Based on an evaluation of current exercise approaches, we look at the key areas of practice that can be improved and benefits that can be gained.

  • By Dave Rouse
  • 20 min read
  • Nov 1, 2022

Crisis Exercises: Are You Wasting Your Time and Money?

Hundreds of thousands of hours and millions of Euros are spent each year planning, executing and debriefing crisis exercises.

The return on this significant investment is rarely fully and honestly appraised. If it were, organisations would uncover an uncomfortable truth; many exercises probably fail to achieve their potential.

This can be seen by even the briefest analysis of lessons from incidents which shows persistent and recurring failures. Lessons from incidents are being identified but not learned from – strongly suggesting many exercises fail to deliver on a core part of their mandate.

This article does not go into the mechanics of designing and delivering an exercise – there are many volumes of guidance for that. Instead, based on an evaluation of current approaches, this paper presents key areas of practice that can be improved and the benefits that can be gained – a Blueprint for Better Exercises.

Exercising In The Context Of Preparedness

All incident and crisis management doctrine includes a preparedness cycle (figure 1) emphasising the principles of continuous improvement and learning from lessons. Exercising plays a critical role in the cycle.

Figure 1 - Example of a preparedness cycle from the US Federal Emergency Management Agency

Why Exercise?

Whilst in reality exercises are used to achieve a range of outcomes, in theory at least the core purpose is to “rehearse and evaluate arrangements, give staff practice in the roles they have been assigned and trained for, and confirm that arrangements actually work in practice” (BSI, 2014 p27).

One of the most important functions of exercises is to facilitate the learning and embedding of lessons identified in previous exercises or incidents and identify new lessons that can be addressed before a future incident occurs.

What Do We Mean By Exercising?

Exercises can take various forms, from very simple to extremely complex. As can be seen from Figure 2, there is a broad but not entirely consistent agreement between guidance bodies on the terminology (and underlying definitions) used to characterise exercise types.

 

International – British Standards Institute US – Federal Emergency Management Agency 
New Zealand - Ministry of Civil Defence and Emergency Management 
UK - Cabinet Office Emergency Planning College International – IOGP-IPIECA

Seminar

Seminar

Orientation

Discussion

Seminar

   Workshop     Workshop
Tabletop  Tabletop  Tabletop Tabletop Tabletop
  Game      
Drill Drill Drill Drill Drill
Simulation  Functional Functional Simulation  Functional
Live Play Full-scale Full-scale Live Full-scale
PD 25666:2010 Business Continuity Management – Guidance on exercising and testing for continuity and contingency programmes Homeland Security Exercise and Evaluation Program (HSEEP)  CDEM Exercises Director’s Guideline  Developing and Delivering Exercises Oil Spill Exercises - Good practice guidelines for the development of an effective exercise programme

Figure 2 – Various taxonomy of exercise types

How do we know current approaches can drive improvement?

Whilst there are many sources of guidance on exercising, there is little quality theoretical or empirical evidence on what makes an effective exercise. An analysis of incidents and resultant lessons clearly demonstrates recurrent and persistent weaknesses, despite exercises.

 

Example: UK Government Response to the Pandemic

The response to the pandemic is an example of many positives – leadership, innovation under pressure and global collaboration, but it also highlights failures not only in exercising but in the risk-specific approach to preparedness. To use the UK as an example, The UK Government “Coronavirus: lessons learned to date” report, published in October 2021 found: 

Previous exercises to test the national response capability, namely Exercises Cygnus and Winter Willow, did not squarely address a disease with the characteristics of covid-19.

— UK Government, 2021 p29

The report acknowledged that the exercises were based on contingency plans derived from the National Risk Register and that the shortcomings of the exercises were caused in part by a focus on risk specificities (the characteristics of the disease), limited consequence analysis (cascading and longer-term consequences) capability and capacity assumptions, complacency and “austerity” which informed the Government’s pandemic preparedness.

The report recommended that:

A greater diversity of expertise and challenge – including from practitioners from other countries and a wider range of disciplines – should be included in the framing of the National Risk Register and the plans that emanate from it.

— UK Government, 2021 p30

Plans for the future should include a substantial and systematic method of learning from international practice during the course of an emergency.

— UK Government, 2021 p30

Example: Multi-agency lessons identified in the UK since 2005

The UK’s Emergency Planning College, part of the Cabinet Office, published a summary of multi-agency lessons learned from 24 exercises and incidents that occurred in the UK since 2005.

It identifies a number of common and persistent trends. Perhaps unsurprisingly to experienced practitioners, communication and situational awareness were the most common themes, both recurring repeatedly over 13+ years.

In its conclusion, the report states:

Where there is a strong pattern of repeated occurrence over time, this probably does suggest that certain lessons are being repeatedly identified by not learned from. This is particularly concerning when they relate to what might legitimately be called basic principles and fundamental good practice.

— Demmar and Leigh, 2020 p10

A Blueprint For Better Exercises

This article presents a Blueprint for Better Exercises. The goal of the Blueprint is to provide a set of considerations and approaches for implementing existing guidance on exercise design and evaluation, based on academic literature and evidence from exercises.

 

Reason for exercise

 

Proactive vs. Reactive

What is the exercise for?

 

Purpose and focus

Who is the exercise for?

 

Team vs organisation vs multiagency 

Who is in the planning team?

 

Diverse experience 

Adjust for variables

 

Facilities, resources, timing, software… 

Exercise type and scale

 

Requirements vs budget

Project planning

 

Ways of working

Alignment

 

A golden thread for evaluation?

Organisation

 

Politics and culture

Scenario development

 

Scenario, injects, assessment 

Influences

 

Human factors

Evaluation and accountability

 

Poor evaluation = evaporated value

Figure 3 - Blueprint for better exercise, Architects: Dave Rouse (OSRL) and Dr El Parker (Independent Consultant)

Factor: Reason for exercise - Proactive vs Reactive

Considerations:

  • Proactive: part of a long-term programme with QA and accountability mechanisms
  • One-off reactive: ad hoc; triggered after change in legislation, new lesson or plan
  • Both demand forensic needs assessment – root analysis cause of previous problems/consequences of change
  • Outline parameters (budget, timescale)
  • Gain genuine support from senior leadership* 

 

Significance:

Establishing an annual cycle of quality-assured exercising and improvement will allow development and improvement to build year on year.

Reactive exercising as a response to change may reap transient benefits but lessons are unlikely to be learned and retained unless exercise forms part of broader learning processes.

Factor: What is the exercise for? Purpose and Focus

Considerations:

Ensure one purpose:

  • Develop and integrate new or revised arrangements
  • Test arrangements
  • Train and develop people’s competence
  • Validate & assure arrangements and competence

One Focus:

  • People vs process vs the whole system
  • Is knowledge development needed prior to practice 

 

Significance:

One exercise can not address multiple needs – no matter the resource thrown at it. Define a series of smaller exercises.

Only competent people can test arrangements.

Only responders with high-level expertise can test and assure complex interdependent arrangements.

Trained and experienced assessors are needed to assess competency

Factor: Who is the exercise for? Team vs crossorganisation vs multi-agency

Considerations:

  • Relate to needs assessment and reason to hold exercise
  • More parties = more planning and more politics…
  • …BUT potential for more value if executed well
  • Evidence of the benefits for key stakeholders*

 

Significance:

Identify needs – Are people competent? do your arrangements work? Is there interoperability across services and agencies?

Response requires effective teams – not just individuals. Crises are often managed by peripatetic/inexperienced teams.

Don’t design the exercise around those available to play – highlight the risks of limitations in those capabilities and capacity. 

Factor: Who is in planning team? Diverse expertise

Considerations:

  • Represent ALL stakeholders
  • Diverse (thinking, expertise, demographic, experience)
  • Break silos and avoid groupthink
  • Team members :
    • Project management
    • Technical advisors
    • Scenario development researchers
    • Evaluation/QA experts
    • Exercise delivery leads

 

Significance:

Diversity builds resilience. Maximise insights which build competence and capability – a diverse development team can bring a breadth of experience, and a depth of understanding and challenge each other’s assumptions.

Don’t overlook second-order stakeholders’/potentially impacted communities’ perspectives.

Factor: Project Planning - Ways of working

Considerations:

  • Collective approach – fidelity & trust
  • In/out of scope – manage expectations
  • Assign responsibilities
  • Milestones and timelines
  • Budget/in-kind
  • Baseline Workshops
  • Understand current capabilities ecosystem – links and interdependencies
  • Define the capacity needed (staff, assets etc) to implement the plan AND what is typically available 

 

Significance:

Planning the exercise is a learning opportunity. The developers of London’s largest exercise (Unified Response 2016) reported the value of shared learning and understanding gained thorough collaborative design (changes were identified during this phase).

The exercise is a project. Adopting project management discipline is critical to success.

Don’t overlook or underestimate the importance of ‘front end loading’ – the robust planning, design and preparation carried out in the early stages of a project’s lifecycle which set the project up for success.

Factor: Alignment - A golden thread for evaluation?

Considerations:

  • Clear statement of need – strategic goal
  • Measurable objectives or learning outcomes
  • Define standards expected for arrangements or competence
  • Align the scenario & injects
  • Capture in an evaluation framework
  • Progress indicators
  • Monitoring & Information Management mechanism

 

Significance:

Align all your elements (the golden thread) – create a “map” that shows how the injects test or evidence competencies or standards and which objectives cover each cluster.

Create measurable performance standards that observers and debrief facilitators can use for evaluation.

 

Factor: Organisations - Politics and Culture

Considerations:

  • Overcome politics, risk-averse culture and fear of losing face
  • Map capability and capacity assessment against credible WCS – be clear on the risk of not stress testing
  • Challenge assumptions (scenario, capability & capacity)
  • Navigating and influencing takes time
  • Many organisations have a preference for exercises that deliver good news…
  • Learning organisations strive for improvement… Good news!

 

Significance:

Too often exercises play safe, the scenarios are within capability and capacity – success is guaranteed! By definition, a crisis is a situation beyond the capacity of those involved.

High-reliability organisations (potential for large-scale risk and harm but balance effectiveness, efficiency and safety) demonstrate deference to expertise, reluctance to simplify, sensitivity to operations, commitment to resilience and preoccupation with failure.

Factor: Scenario development - Scenario, injects, assessment

Considerations:

For scenario:

  • Don’t make it up – define evidence-based credible worst-case scenarios (WCS)*
  • Resist pressure to deviate

For injects:

  • How much role-player free-play?
  • Realism – technical expert actors

For assessment:

  • Align injects to performance standards indicators
  • Provide individual/team feedback using competence statements & confidence ratings
  • Agree on predefined feedback statements – “working towards” rather than “needs improvement”

 

Significance:

Avoid playing safe – by researching case studies, applying predictive models, and applying systems thinking (Checkland 1989 & 2011) to define a credible worst-case scenario that details hazard/threat – impacts – consequences.

Look beyond the risk register when developing credible WCS. The risk register is not always imaginative enough. Consider not only what could credibly happen but the paths that could lead there.

The organisational culture may mean teams make safe assumptions – use objective experts to challenge decisions and free-play reactions during exercises.

Use the performance standard response to an inject as the basis of feedback to the team.

Factors: Influences - Human Factors

Considerations:

  • Understand perceptual biases through consultation e.g. assumed effectiveness of risk controls
  • Test confirmation, anchoring & availability bias via tailored injects
  • Test procedural vs adaptive expertise
  • Provide the evaluation framework (standards expected) to participating organisations/ delegates 

 

Significance:

Don’t play safe - design exercise scenarios that challenge nontechnical skills (Flin et al 2008).

Areas of weakness commonly stem from poor situational (risk) awareness, unwillingness to deviate from planned actions despite evidence to the contrary, and sticking to failing procedures in times of cognitive overload.

Tabletops with breakouts for discussion provide a safe space for analysis and feedback.

Resilient practitioners/responders demonstrate metacognitive expertise – they know when they can't solve a problem and adopt effective (collective) strategies.

Factor: Evaluation & Accountability - Poor evaluation = evaporated value

Considerations:

  • Train a diverse range of evaluators from inside & outside the sector
  • Develop a better debrief than ‘three things that went well, three things to do better AND provide debrief training for delegates
  • Use the evaluation framework (standards expected)
  • Establish a moderation process for consistent evaluation
  • Governance process needs to ensure lessons are actually implemented across teams/agencies – track accountability
  • Measure, evidence & report impact and ROI
  • Publish transparently 

 

Significance:

Ensure everyone follows the golden thread, and evaluate what the exercise intended to achieve; not what was salient during the debrief.

Empowering people to contribute in a debrief – to identify areas of weakness – requires an understanding of performance standards and trust that contributions will be used ethically, and belief that positive change will result.

Assign ownership to each lesson, agree on appropriate actions and ensure the quality cycle demands evidence of change and retention.

Factor: Adjust for variables - Facilities, resources, timing, software…

Considerations:

  • Less is usually more
  • Exercise management software has pros and cons
  • Tendency to go big, but several shorter activities may be better for retention, developing dynamic risk decision-making, confidence…
  • Resist the use of tools and templates that restrict thinking/decision-making. 

 

Significance:

Exercises are not a showpiece, no organisation is invincible.

Expertise and resilient arrangements take time and practice (Ford & Schmidt 2001).

Competence has a half-life.

Aide Memoires are useful but ensure they drive creative/adaptive rather than proceduralist thinking.

Factor: Exercise type and scale - Requirements vs budget

Considerations:

  • Balance between time/cost/quality
  • Purpose (teaching vs deliberate practice vs testing vs performance demonstration)
  • Focus (people vs process vs the whole system)
  • Where it fits in the programme/recency of previous exercises
  • Competence of participants vs gaps
  • Maturity of arrangements vs gaps.

 

Significance: 

With a diverse team and a bit of time, it is possible to create focused, challenging and beneficial exercises.

Having an exercise programme (a variety of types/scales) with a quality assurance mechanism will reinforce lessons that need to learn, embed changes, and save money in the longer term.

Strategic Benefits of this Approach

Too often, exercises play safe. They use scenarios that are within existing capability and capacity, and designed to finish within a neat eight-hour window. 

Success is virtually guaranteed. 

Exercising routine arrangements like this are a start and may have a role in developing muscle memory and mitigating skill fade. However this will not prepare teams for complex incidents and crises, which by definition are unpredictable, do not go to plan, and threaten to exceed the capability of those involved.

Thoughtful application of the approach outlined in this paper will provide strategic benefits to any organisation – improving preparedness and improving the return on exercise investment.

A programme of thoughtfully-sequenced, well-defined small-scale exercises, complemented by guided self-study, will develop and embed the knowledge required by responders. The programme is repeated with new cohorts of responders, evolving where appropriate. This approach is sustainable, efficient and agile. 

This allows large-scale exercises to focus on the most complex areas, delivering maximum benefit from the investment they entail, and to take resilience to the next level. Using expert-informed, credible worst-case scenarios, large-scale exercises can genuinely challenge participants and stress-test interdependencies present in complex organisational

systems. This is only possible when the focus is clear and the fundamentals have been developed progressively beforehand. 

Exercising in this manner, responders develop the mindset necessary to work effectively under pressure, expand shared mental models of system dependencies, and build the confidence needed to perform at their best when the worst happens. 

In addition to improving response readiness, organisations with high crisis competence develop a resilience mindset and improved risk management culture. Staff who are confident, adaptable, quick thinking, agile and collaborative, who trust each other from having worked together under pressure, can apply worst-case-scenario thinking to business

as usual challenges, and are better equipped to anticipate and manage unforeseen consequences.

Conclusion

If the pandemic has shown us anything, it is that we must plan for uncertainty and develop capabilities to deal with it.

Organisations already have the capability to critically assess needs and gaps, develop performance standards, and consistently evaluate competencies and arrangements.

The main shift most organisations need to make to improve the return on investment of exercises and realise the full preparedness and strategic benefits outlined above is cultural.

The shift is from a culture where exercises are ad-hoc, standalone projects towards one where they form part of a much broader quality and improvement cycle.

By integrating exercise planning, delivery and evaluation phases into wider quality governance systems, organisations will benefit from improved readiness, a more effective response, swifter recovery and strengthened resilience.

References

British Standards Institution, 2014. BS 11200:2014 Crisis management – Guidance and good practice. London.

Checkland, P. B. (1989). Soft systems methodology. Human systems management, 8(4), 273-289.

Checkland, P. (2011). Autobiographical retrospectives: Learning your way to ‘action to improve’ – the development of soft systems thinking and soft systems methodology. International Journal of General Systems, 40(05), 487-512.Gunns, J. and Leigh, M., 2016. Developing and Delivering Exercises. UK Government Cabinet Office Emergency Planning College. York.

Demmar, D. and Leigh, M., 2020. EPC Lessons Digest – Summary of Multi-Agency Lessons Identified in the UK Since 2005. UK Government, Cabinet Office, Emergency Planning College. York.

Flin, R., O’connor, P., & Crichton, M. (2017). Safety at the sharp end: a guide to nontechnical skills. CRC Press.

Ford, J. & Schmidt, Aaron. (2000). Emergency Response Training: Strategies for Enhancing Real-World Performance. Journal of hazardous materials. 75. 195-215. 10.1016/S0304-3894(00)00180-1.

Gunns, J. and Leigh, M., 2016. Developing and Delivering Exercises. UK Government Cabinet Office Emergency Planning College. York.

IOGP-IPIECA, 2014. Oil spill exercises – Good practice guidelines for the development of an effective exercise programme. London.

London Fire Brigade (LFB) 2017. Exercise Unified Response Evaluation Report.

New Zealand Government, Ministry of Civil Defence & Emergency Management, 2019. CDEM Exercises Director's Guideline for Civil Defence Emergency Management Groups [DGL 10/19]. Wellington.

US Government, Department of Homeland Security, Federal Emergency Management Agency, 2020. Homeland Security Exercise and Evaluation Program (HSEEP). Washington DC.

UK Government, House of Commons Health and Social Care and Technology Committees, 2021. Coronovirus: lessons learned to date. London. 

Contributor - Dr. El Parker

Dr El Parker is an independent consultant specialising in risk, resilience and society. She has over 22 years of experience working with the public, third and private sectors in risk assessment, emergency preparedness, risk reduction and recovery.

She is passionate about improving policy and systems to ensure locally appropriate, people-centred interventions.

As Associate Professor for Disaster and Emergency Management at Coventry University, and since leaving academia, El works with the UK and international governments, blue light services and private companies.

Her work helps organisations develop their capacity to work effectively and accountably with those most at risk, embedding the latest academic thinking into professional practice.

elpconsultingrers@gmail.com