As part of our purple teams program here at SRA, we develop many different threat simulation plans (“bundles”), such as our Threat Simulation Indexes. To support the creation of these bundles, we developed the “Market Maker” (“MM”) suite of tools. Market Maker is a Python library, collection of command-line scripts, and several extensions. In this blog, we will discuss some of the lessons that led to functionality of the Market Making tooling. For a more comprehensive overview of the functionality, refer to the documentation in the repo here: https://github.com/SecurityRiskAdvisors/marketmaker
Overview
At its core, Market Maker very simplistically merges test cases documents (YAML) into a singular bundle of test cases for use in testing exercises. The generated bundle can then be imported into other tooling, such as VECTR. As an example, our public Indexes use the Market Maker test case and bundle formats:
(Financial Services Index 2024). [VIDEO] Over time, the tooling has expanded based on requirements as well as some growing pains.
Content should exist on its own
Before Market Maker existed, attack plan content development happened directly within our VECTR platform. Users created their test case in a non-production instance then shared the exported STIX templates for use on projects. This was not ideal for a host of reasons, namely:
- The development was intimately tied to VECTR. The templates were stored in complex STIX JSON files and any changes to template required either an active VECTR instance running or manually editing the STIX JSON.
- The STIX JSON was also not conducive to casual viewing. If a person was interested in quickly reviewing the contents of a template, they couldn’t without first loading the template into a VECTR instance.
- Having concurrent contributions to a bundle template was infeasible since there was no central authority for template definitions.
At the time however, we were maintaining a single bundle that only received updates a few times per year, so any issues were ultimately minor. As we started to expand our bundle offerings though, these issues became more pressing. This became the foundation of Market Maker. Test cases would be stored on their own in a library for reuse in multiple bundles. Then, each bundle would become a list of those test cases. Market Maker could then generate an export from this information. These definitions were also moved into a Git repository. Contributions to templates could be managed through pull requests by any number of users and a record of the changes tracked in the history.
Operator guidance
Performing a test case often has requirements beyond just the command/action. This could be a payload, a piece of infrastructure, or some specific target configuration. It also isn’t always desirable to include this in the test plan either since it covers information relevant for pre-execution. In order to both support more complex test cases as well as inform downstream users (e.g. internal red team operators, public bundle users, etc), we created “Operator Notebooks”, which are structured Markdown documents for a bundle. Each test case that requires additional information related to one of the following areas could have operator guidance included in such a notebook: 1) prerequisites, 2) execution, 3) cleanup, 4) additional notes/considerations, and 5) external resources. For an example notebook, see: Operator guidance also exists independent of test case definitions so that test cases can re-use operator guidance, similar to how bundles can re-use test cases.
Keep things DRY
If you look at Tactics, Techniques, and Procedures (TTPs) as a gradient of specificity, where Tactics are generic and Procedures are highly specific, a Variant is somewhere between a Technique and Procedure. Consider dumping lsass.exe process memory. This activity would fall under the “Credential Access” Tactic and the “T1003.001: LSASS Memory” Technique per MITRE ATT&CK. An adversary has many options for performing this credential dumping activity, such as using mimikatz or Sysinternals’ ProcDump. Each of these options is a Variant and each specific method of using that Variant is a procedure. A library that houses the test cases used by Market Maker is organized based on this paradigm; specifically, three properties of the test case are used for the organizational hierarchy: 1) the MITRE technique ID, 2) the variant of that technique, and 3) the version of that variant. Using procdump.exe to dump lsass.exe would be stored in the library as: [PIC 1] Each individual Variant can have any number of versions that differ slightly from one another. In some cases, these differences are not material to the test case but nonetheless require new test cases. One notable side effect of this approach is the occurrence of sprawl, where you end up with many copies of a single Variant. To reduce the amount of Variant sprawl, two features were added to Market Maker. The first is “partials”, which are canned statements for the block and detect section of the test case. Rather than directly placing the guidance in the test case, it can be placed in a centralized document alongside other guidance then referred to in the test case. This provides the benefit of creating a single source of truth for block/detect wording, limiting variations of the same ideas for block/detect wording, and making updates to the wording easier as it only needs to be performed once. [PIC 2] The second feature is to allow the bundle definitions to override test case fields on a limited basis. For example, rather than maintain one password spray test case for a specific web platform used in one bundle then duplicate that as needed for each different platform, you can maintain a generic test case then have the bundle override the test case name, leaving all other fields the same. This limits the overall number of test cases that need to be maintained individually and ensures consistency among similar test cases. For example, if you need to specify a target for a test case, you can do so for a specific test plan by overriding the name rather than changing or duplicating the source (shown below). [PIC 3]
Too much metadata & Extensions
Each new feature added to Market Maker typically required additional metadata on the test cases. When adding guidance to a test case, the guidance identifier(s) was added as metadata. When adding detection logic to a test case, the detection identifier(s) was added as metadata. Eventually, the test case metadata section ballooned in size and would have likely continued to do so. This was untenable as it would greatly hurt the readability of the test case as most metadata being added was not intended for human use but rather for use by tools. To fix this issue, we looked no further than the aforementioned “partials” implementation. Rather than place all metadata in each test case, we could have the mappings exist in a single file for each use. In so doing, the tools could pull the relevant information without the end consumers needing to see it while also making it easier to have a single view of all mappings across the library (rather than needing to view each test case individually). These changes also came alongside Market Maker extensions. As the features expanded, the tooling code naturally needed to expand to support it. To support expanding use cases without needing to modify core code, a mechanism for extending the tooling’s functionality was added. With this public release of Market Maker, any user wishing to hook into the functions of the library can do so without needing to fork or otherwise modify the core code. Market Maker is being released with the following first-party extensions:
- Graphviz: Generates a spider-web style graph visualization for technique IDs to associated groups based on the Blueprint data.
- Guidance: Generates Operator Notebooks based on a mapping of Variants to guidance documents
- Partials: Centralized block and detect language management for Variants
- Sigma: Generates detection documents based on a mapping of Variants to Sigma rule IDs
- VECTR: Modifies Variant data for use with VECTR
Closing
We are releasing the Market Maker tooling publicly on our GitHub here: https://github.com/SecurityRiskAdvisors/marketmaker. We also plan to continue committing development resources to this tooling for the foreseeable future for both maintenance and new feature development. And while a lot was covered in this post, it is not the full functionality of the tool. Full details of how to use the tool and its features can be found in the repo here: Docs. For a quickstart guide, see: Quickstart. For questions/concerns, feel free to reach out to me directly @2xxeformyshirt.
Evan Perotti
Evan specializes in network penetration testing, web application security testing, open source intelligence gathering, and security testing process automation.
He has experience in a variety of industries including retail, insurance, financial services, and healthcare.