Using Tabletop Exercises to Strengthen OT Security Maturity

by | Feb 7, 2023

If you are part of an organization grappling with how to secure your Operational Technology (OT) environment, you are not alone.

Let’s start with the basics:

  • Do you have an Incident Response Plan (IRP) that includes your OT environments and assets? Hint: the enterprise Information Technology (IT) IRP won’t cut it.
  • When was the last time you tested your IT Incident Response Plan?
  • More importantly and germane to this post, when was the last time you tested your OT Incident Response Plan – assuming you have one.
  • Do you have a definition or list of what you consider OT assets? Check out some thoughts on ‘defining OT’ by my coworker, Jason Rivera, here: https://sra.io/blog/what-is-ot/.

In these environments with high availability needs and often antiquated equipment, it can seem daunting finding your roadmap to operational security. A Tabletop Exercise (TTX) can test critical components of an OT IRP without risking downtime or unexpected results in a sensitive production environment.

A Brief History of OT Security

OT is the system or systems that monitor or control production equipment, assets, processes, and events. OT is often found in critical infrastructure and is increasingly under attack. According to the 2021 Internet Crime Report, there were 649 complaints last year from organizations in the critical infrastructure sector that were victims of a ransomware attack. So, if you’re one of those asking yourself how to protect these environments, you’re in good company.

For organizations with an OT environment, the cost of a cyber-attack is considered in much more than dollars and cents. Plant safety regulations are often referred to as “written in blood” because when things go wrong, these environments can be dangerous for both operators and downstream customers. Attacks on the OT environment aren’t new; Stuxnet was unique when it came to light back in 2010 but it brought forward a broad awareness of cyber-attacks weaponizing risks to the cyber-physical space.

Since Stuxnet, attacks have been escalating:

Timeline of Notable ICS/OT Cyber Attacks, more information in Appendix

Why OT Security Matters

OT environments have unique considerations and in 2022, the cost of a ransomware attack in critical infrastructure climbed to a historic $4.82 million USD, according to the IBM & Ponemon Institute research data:

  • 27% of companies experienced a destructive or ransomware-specific attack
  • 57% of companies state their OT environment is not in a state of cyber readiness, and
  • 55% of organizations report that they are not confident in their ability to minimize the risk of cyber exploits and breaches

That means less than half of companies with an OT environment feel able to protect it properly. Prior to Stuxnet, measures like air gapping networks could be seen as providing some protection, but as these environments have increasingly been benefiting from IT/OT convergence, threat actors have been making use of those connections to infect and exploit.

Benefits of Tabletop Exercises

Tabletop Exercises (TTX) can be an effective way for organizations of any size or maturity to test their current process and identify enhancement opportunities specific to their environment.

  • A Tabletop Exercise is an informal, discussion-based exercise with teams reviewing their anticipated roles and responses during a specific situation, prior to any incident taking place.
  • As clear, critical thinking is often the first to exit the building during a crisis or other high-pressure scenario, setting aside several hours to evaluate and test IR plans regularly can give companies a huge advantage.
  • Running a TTX can help reveal problematic or missing steps in the IR plan, conflicting perspectives from team members, or even a fundamental lack of understanding.
  • A TTX is NOT an audit, there are NO wrong answers and there are NO penalties of any kind.
    • The goal is to foster a productive discussion to find any gaps and enhancement opportunities in the current IR plans, processes, and playbooks.
    • It is also an opportunity to check how current processes respond or can be adapted to changing world events- consider how many companies had a pandemic response plan in 2019 compared to today.

Tabletop Exercise as a First Step to Improving OT Security

With these facts in mind, companies within the OT space are increasingly looking for ways to protect their environments. Enter the humble yet effective Tabletop Exercise (TTX) – no matter where your organization is on your strategic roadmap for OT security, a TTX can be an effective evaluation tool to check your current security posture and find enhancement opportunities.

In fact, the topmost effective ways to mitigate the cost of a security incident are:

  1. Having (or forming!) a cross functional incident response (IR) team
  2. Having an incident response plan (IRP) specific to the OT environment
  3. Testing that IRP regularly

On average, organizations that follow this guidance gain a cost reduction of up to 58% during a data breach, to the tune of up to $2.66 million USD saved.

Differences between an IT and OT Tabletop Exercise

Many business executives have at least some familiarity with Tabletop Exercises, with Disaster Recovery and Business Continuity Planning being two topics frequently covered. For cyber incident response exercises, there can be a difference in fundamental thought processes between IT and OT that needs to be considered.

  • IT cyber security concerns typically align with the C.I.A triad- where confidentiality, integrity, and availability of data at rest and in motion is the mission.
  • OT objectives often follow the SIAC quartet, which focuses on safety, integrity, availability, and then, finally, confidentiality.
  • This means that OT operators and managers have an entirely different set of priorities, they are concerned with the physics of moving machinery, with the safety of human life being the mission critical objective.

Comparing the two missions illustrates why the two groups can feel finding common ground is difficult. Merging these groups together to talk through a hypothetical situation can lead to better understanding of each other’s roles and can help foster a spirit of collaboration rather than competition.

Best Practices for Building your Tabletop Exercise:

Regardless of your current organizational OT security state, there a few things to consider while architecting your TTX to ensure success:

  1. Identify the goal – Focus your scenario on a specific area or topic that you want to test. Communicate goals as part of the exercise:
    1. Are you testing your company’s ransomware response? Stress-testing a newly deployed technology? Training new team members? Is your exercise compliance driven? Are you testing cross functional team collaboration and cooperation?
    2. Whatever topic you tackle, make sure the end goal is well understood
  2. Encourage internal conversation in a judgement free zone:
    1. It should be made clear there are no wrong answers, provided they are relevant and honest
    2. Have observers present to take notes not just on what was said, but also if there are extended pauses or non-verbal indications of confusion or misunderstanding from the participants
  3. Keep the conversation flowing – keep an eye out for signs of frustration or conflict:
    1. These kinds of conversations can help build and strengthen cross functional communication
    2. While it is important to note areas of enhancement opportunities, this is not a forum to air grievances or rants
  4. Collect key takeaways – the goal is to find the areas of strength and observations on areas where enhancement opportunities exist
    1. Praising observed strengths can help cement the idea that these are collaborative, not adversarial exercises and increase participation
    2. After-exercise deliverable should also include suggestions and next steps to apply enhancement opportunities in a timely manner

Levels of OT Security Maturity:

No matter where you find yourself on the roadmap to OT security maturity, there are ways to structure your Tabletop Exercise to help move the needle on forward progression.

1. Foundational:

Organizations in the earlier stages of their cybersecurity journey are often adding the foundational controls needed to protect their OT space. Things like isolating OT from the corporate network, having a separate industrial-protocol-specific firewall between the two areas, automating local backups and storing a copy offsite, and ensuring secure remote access for manufacturers or vendors are all important beginning steps in an OT security roadmap.

Potential teams to include: Management, Cybersecurity, OT and IT Network Architects, Key Vendors or outsourced teams like Managed Service Providers

Possible discussion points:

  • Can your backups be validated, and can your critical devices be restored in the event of an emergency?
  • Who or what role would detect the kind of incident that would require shutting down a machine, or disconnecting one machine, a line, or the entire plant from either the intranet or internet?
  • Is there documentation on who or what role would authorize shutdown or disconnection and how that would be communicated companywide?
  • Is there a contingency plan if the person or role designed to detect or authorize action in an adverse scenario were to be unreachable?

2. Intermediate:

At the mid-level of maturity, we see companies implementing an industrial DMZ layer, a separate Active Directory domain, collecting logs in a security incident and event manager (SIEM), scanning removable media prior to connecting to the OT environment, and application whitelisting. Many companies here use TTX to look for gaps – be they in logging methods, data source collection, network segmentation changes, access control issues, security or safety measures, or communication processes between cross functional teams.

Potential teams to Include: Engineering, Cybersecurity, Management, OT Operators, Compliance, after-hour/on-call coverage teams

Potential discussion points:

  • What is the process to ensure only non-infected removable media is used in the OT space?
    • What is the validation and approval process prior to removable media usage?
    • Who or what role owns that process?
  • What does “known good” look like in this environment (to help spot and determine malfunctions or signs of infection)?
    • How is a “known good” baseline determined and captured?
    • What is the process if a device starts operating outside of documented baseline parameters?
  • Are there compliance regulations to consider?
  • Who or what role handles after-hours monitoring and escalations?
    • Is there an after-hours approval process documented?
  • What is the determination process for applying security patches?Can manual, paper-based plans be used if some or all electronic systems are hobbled or unavailable?

3. Advanced:

For companies further along their security maturity journey, regulatory compliance starts becoming more mature, and we start seeing companies that use security orchestration, automation, and response (SOAR) and security operations centers (SOC) to monitor alerts and network traffic. Other more advanced testing capabilities can be applied with more confidence, such as red or purple team activities. At this point on the roadmap more advanced concepts should be discussed, and this is the stage where specific OT IR playbooks can and should be tested to try and identify any gaps and ensure all included parties are in alignment.

Potential teams to include: Management, Cybersecurity, SOC Analysts, OT Operators/Managers, MSP representatives if outsourced, OT and IT Network Architects, Engineers

Potential discussion topics:

  • If security monitoring is in place and outsourced, do you understand how alerts will be escalated and is everyone in agreement that the correct stakeholders are included?
  • If security monitoring is in place and handled internally, is there sufficient staff with the correct OT and cybersecurity background needed?
  • How long could a shutdown/disconnection be in place before a negative business impact would be seen?
  • IT is the largest attack vector and surface for OT, so testing the ability of SOC staff to:
    • Know where and what constitutes the “crown jewels”, or the devices most important to business function
    • Know where any pivot points exist between IT and OT environments, and have a baseline of what “known good” communication looks like between the two
    • Be able to track and isolate an attack that is originating from IT and oriented towards OT

Conclusion

Operational Technology (OT) refers to the systems that monitor and control production equipment, assets, processes, and events; it is often found in critical infrastructure, and it is increasingly under attack.

As organizations look to advance their state of OT security, performing Tabletop Exercises is an effective way to test the people, processes, and plans prior to a predicament. Find out more about how SRA can help secure your cyber-physical environment here.

Having and regularly testing an IR team and IR plan are the most effective ways for companies to lower their costs in an actual emergent situation. Conducting regular Tabletop Exercises can help identify weak points in current plans or processes, foster cross functional team understanding and collaboration, and proactively work to protect your OT environment.

 

Appendix & Additional Resources

Mandie Grosskopf
Senior Consultant GRID, CySA+, Security+, Network+, Security Analytics Professional | Archive

Mandie has a strong networking background, with a passion for network security monitoring (NSM) and cyber-physical systems (CPS). She has spearheaded the design and enhancement of Incident Response (IR) processes for clients in multiple vectors, specializing in using Tabletop Exercises (TTX) as an improvement tool.

Previously, Mandie was the Manager of a SOC at an MSP, and she brings her years of experience leading a team to protect IT data to her work expanding IR coverage to the cyber-physical space.

Mandie currently holds her GIAC Response and Industrial Defense (GRID), Network+, Security+, Cybersecurity Analyst+, and Security Analytics Professional certifications.  She holds a BS in Information and Communication Technologies from UW-Stout.