You are likely going to execute a TTX or two this year. How you execute matters. This playbook on facilitating an effective TTX can help lead you to success (even with the most skeptical C-Suite).
Start with the why.
Why are we spending the time to do this? Let’s break it down into four categories.
- Money: Starting with the obvious. Incidents and breaches cost money. Not just the ransom demands for cryptocurrency, but the dollars spent engaging with your cyber insurance broker, incident responders, forensics teams, fulfilling regulatory fines or reporting obligations, offering potential remuneration to customers and the opportunity cost of dealing with the incident instead of focusing on growth and innovation. Being efficient at responding to incidents significantly cuts down on the cost.
- Resilience: The tools are in place, the Security Operations Center (SOC) has its playbooks, security awareness training is on lock, but still incidents happen. An Incident Response (IR) team with an unpracticed and impractical incident response plan will waste valuable time that could be spent kicking the threat actors out of the network.
- Reputation: Incidents happen all the time, but it is a breach that will impact your reputation. Knowing when to call it a breach, how to address it with employees, the press, and the public are the difference between losing business because you appear uncoordinated and unprepared or showing confidence in the face of adversity.
- Muscle Memory: “We talkin’ bout practice?” (I’m from Philly and the Sixers Allen Iverson years were special). You don’t shoot hoops for the first time in a couple years and expect nothin’ but net. Likewise, you shouldn’t expect to respond to an incident without significant damage unless you have practiced the response.
Prepare
- Understand the current state: What are the results of the latest purple team? How did the red team gain access? Figure out if an incident response plan exists. Don’t scramble to update it or create it before the TTX. Use the TTX to drive the updates.
- Set the date and location and confirm participants: Make sure the right people are invited and confirm their attendance. Try to make it in-person, but if not, work out the tech ahead of time. Follow-up multiple times prior to the date. Bring a note-taker to capture all the talking points and help generate the after-action report.
- Build a believable and impactful scenario: Do your research, understand the organization and what really impacts the business. Use that to build your scenario and be specific. Saying “Your mission critical applications go down” has significantly less impact than “PaymentAppX can no longer process customer payments and the sales team is taking orders by writing credit card numbers on sticky notes”.
- Know your audience: The C-level isn’t going to know how to mitigate the threat and SOC analysts don’t decide whether or not to pay the threat actor. Don’t pretend that they do. I typically execute the “technical” TTX first and use the results of that exercise to inform how the “executive” and “business” TTXs are built. This can give you a good idea of what the technical team expects of their executive team and vice versa. Here’s how to tackle your most common team scenarios:
- Executive Level: Focus on what they will most likely be asked during a real incident. Questions like “Should we pay?” and “Who talks to the press?”. Executives (likely) do not know how to quarantine a host or rotate the KRBTGT tickets. They should think about impact from a business legal, regulatory, and communications perspectives. Hint: [SA1] representatives from those group should be in attendance.
- Technical / SOC / CIRT Team: Conversely, don’t pose questions to the technical team that distract them from their mission: eradicating the threat. Consider a choose-your-own-adventure format for this crowd, it allows them to think through their decisions and understand the pros and cons of their actions. For example, “if you rotate the KRBTGT before you reset users’ passwords, how can a threat actor maintain access?”
- Business Team: The business generally wants to know how they can get back online and how quickly. They also need to answer questions like “What’s the impact of taking these systems offline?” and “What are the downtime procedures?” and “Who are our critical vendors for your business unit?”.
Execute
- Be realistic.
- Technical balance: Know what tools are in play, what types of systems are in the environment, and how the network is architected (cloud heavy, on-prem, hybrid). You don’t need to know every detail, just enough that the attack path resonates with the audience. There may be a person that calls out to say, “You can’t do that on our network!” That’s ok, address this early on, and remind them that the attackers have no agreed-upon scope.
- Mimic industry specific real-world attacks: Ransomware is getting a bit old for those that have been doing this for a while. Attackers constantly change TTPs and your TTX scenario should reflect the types of threats that organizations are currently facing.
- Let THEM talk: Play a true facilitator role and be comfortable with uncomfortable silence. Let the team in the room (or call) talk through salient points and discover the gaps in their processes.
- Be engaging.
- Polling and guiding questions: Polls are a crowd favorite for any scenario and they help keep everyone engaged and focused. Try to avoid clearly “right and wrong” answers, but instead make them difficult decisions or questions that invoke additional conversation. Make sure the polling system is accessible from any device (phone, tablet, laptop) and keep them anonymous. No one wants to be called out for being the dissenting voice that thinks “hey, maybe we should pay that ransom” (turns out, they might be right after talking it through).
- Set expectations: Be clear on what you are there to accomplish and what everyone’s role is. It is ok if folks are hesitant to participate at the start. I always remind them that the TTX is the best time to clarify things and ask the “obvious” questions – or even admit “I don’t know”. Ironing things out now saves valuable time during real incidents.
- Keep the pace moving: Remember, you are condensing a real attack timeline into a short window. It’s easy to go down rabbit hole and focus on a single area. Note where the team needs more discussion and keep them moving onto the next information inject.
- Be entertaining.
- Use visual storylines: Slides with just text are as boring as they sound. Use screen captures from real systems and applications on the network. Pre-record videos and use deep fakes. Use your experience in real incidents to describe what’s happening to the audience.
- Tell stories (real ones): Being able to provide credibility to the scenario by telling a similar, real-world story is both informative and entertaining. Careful not to divulge things that might be sensitive.
- Be relatable: You will lose the audience quickly if you are not talking at their level. I often use a slide like this one below to demystify the often way too technical IR plan:
Recap and Improve
- What did we learn?: Don’t ask them to name everything in the moment, but just think of one thing they are going to take back to their team to work on. This is easier to grasp in the moment and leaves them with something immediate and tangible while the broader report is generated.
- Improvements: Generate a report that focuses on real improvements. Note short-term and long-term fixes to processes and procedures. Use examples from the TTX to back up the observations. The SANS PICERL model is still a good way to organize your TTX observations: https://www.sans.org/media/score/504-incident-response-cycle.pdf
- Use the results to inform your yearly security activities:
TTXs should be more than a necessary exercise for your organization. Whether you run them yourselves on call or have an external organization like SRA to help facilitate, they should stoke conversations that lead to meaningful improvement, role clarity, and help make a real incident a little less painful.
Chris Salerno
Chris leads SRA’s 24x7 CyberSOC services. His background is in cybersecurity strategy based on NIST CSF, red and purple teams, improving network defenses, technical penetration testing and web applications.
Prior to shifting his focus to defense and secops, he led hundreds of penetration tests and security assessments and brings that deep expertise to the blue team.
Chris has been a distinguished speaker at BlackHat Arsenal, RSA, B-Sides and SecureWorld.
Prior to Security Risk Advisors, Chris was the lead penetration tester for a Big4 security practice.