VECTR Fundamentals: Why Execute Test Cases?

Most agree, Purple Teaming is an effort that can deliver a high level of value to their organizations. However the nuances of an organization’s approach in Purple Teaming can have a large impact on the effectiveness of their program. Competing teams inside a cybersecurity department can have different incentives and goals, making deciding a starting point challenging. SRA has developed a results driven approach with VECTR, and it all starts with a concept we call a Test Case.

A Test Case is an individual test to be run within a campaign. A Test Case includes a set of commands or instructions and any necessary accompanying data designed to help a Purple Team operator perform a specific, repeatable security testing activity. Once a Test Case is performed, VECTR allows you to capture additional information like when the test was performed and if it was detected by defense tools.

In the context of the MITRE ATT&CK Framework, a Test Case represents a specific, repeatable instance of an attack technique. A Test Case Template may be considered similar to a “Procedure” in the MITRE Enterprise ATT&CK framework. Different Test Case variants can map to the same MITRE ATT&CK technique ID.

Let’s break down what a Test Case is with an example.

1. Attack Status & Time – Using the buttons in the UI or via clicking the gears you can specify when attacks were started, paused, stopped, or abandoned. This is valuable for the Blue Team to track down logs and find correlating alerts.
2. Assets – You can log any Targets/Sources here. IPs, Account Names, Machine Names, any value you would find useful in tracking logs or filtering/reporting on in post-execution.
3. Red Team Details – This is the specific operator guidance provided to the Red Team. Red Team Operators should be following this guidance specifically to ensure consistency in evaluating tool behavior.
4. Automation/Payload Generation – VECTR Supports generating payloads that can provide logs and upload back to VECTR. This is an optional functionality which will be covered in more depth later this year, to learn more about this see our documentation here.
5. Defense Activity – Here you can see our “Per Tool Outcomes” we introduced last year. The goal is to log each individual Defensive Tool and it’s involvement in the given Test Case. The Test Case screen evaluates these to populate the overall Test Case Outcome seen in the top right.
6. Outcome Notes – This is where the Blue Team gets to tell their story. What alerts were triggered or not triggered? What systems were checked? Feel free to link out to tickets in other platforms to fully document the investigation.
7. Tags – Tags are one of the most powerful features in VECTR. Allowing users to create their own attribution in any way they see fit. These tags can be used to filter out or exclusively include in the reporting engine.

The benefits of defining and leveraging this specific operating model are many.

1. Specificity ensures confidence in evaluating outcomes. Often when security testing, offensive engineers are focused on “winning”. Not necessarily testing specific operator guidance for defense effectiveness.
2. Repeatability allows tracking outcomes in repeat testing, allowing you to track trends. Showing improvement over time is a key piece in building Purple teaming visibility.
3. Documenting your Test Cases into templates helps operationalize your Purple Teams. The learning curve to building out your Purple Team operations is a real barrier, leveling up your team through the process is a key to success.
4. Part of the maturity process of testing specific scenarios is raising the capability and effectiveness of both Red and Blue contributors. We find that both teams involved often gain insights and satisfaction from seeing the other side at work.

Building out an operational model that implements and executes test cases regularly is a key to measuring your threat resilience. With this foundational piece in place, teams can mature into regular testing patterns. This operating model builds trust in the validations recorded, and provides quality data for you to report on across your organization and upward to management teams. Truly building a threat resilience picture you can have confidence in and communicate clearly.

It’s important to attempt to truly understand the end goal, to quantify your organization’s preparedness against various threats that they may face. The more scenarios you validate through using Test Cases the richer your reporting becomes, more complete your MITRE Heatmap report will look, the better conversations you can have with stakeholders around your vendor/tool efficacy. All of these things help the security function of your organization move forward. We will be covering reporting in a subsequent blog post later this year.

Make sure to check back in next week, where we’ll be revealing our new “Test Case Panel 2.0” update along with how to access the preview!