If you really need one, here’s the TL;DR – The SBOM market is emerging. Asset owners are unsure if they want them and suppliers/OEM’s are either considering adoption, have already adopted, or have a “shoo fly” mentality (for now). The “SBOM Challenge” was well intended but could have, and should have been more comprehensive. It faced an identity complex ranging from competition, to challenge, to demo, and then back to challenge. But at its core, we believe that is more reflective of the state of SBOMs than any one particular person or entity’s capability.
Having said that, if you needed a TL;DR then this post might not be for you. We’re going to go deep into SBOMs and provide a pure take on the entire challenge. Strap in or get out now.
Challenge Overview
S4x23 brought in ~1100 attendees to the Loews hotel in Miami and delivered content over the course of the week (Monday-Thursday) on many relevant topics and included challenges, talks, networking events, and more. For us, it began bright and early Monday morning when we joined 4 other challengers in the SBOM pavilion.
The challengers were all granted access to 3 artifacts curated by Idaho National Laboratory (INL) for analysis and were to deliver a component inventory for each (SBOM) in a standard, machine ingestible format. They were identified as follows:
- A Windows MSI file that we identified as OSIsoft’s PI ProcessBook
- A piece of “baremetal” firmware that we identified as targeting the ABB TPU-2000R Transformer Protection Unit
- An INL-built OpenWRT firmware targeting a Meraki MR26 wireless access point
By the end of Day 1, SRA submitted our 3 generated SBOMs from the FACT tool in CycloneDX format, as well as our “vulnerability enhancements” as part 2 of the challenge. On Tuesday, our third and final challenge task was to ingest VEX documents created by INL and demonstrate how we would make use of that information to further enhance vulnerability management.
The challenge was entirely performed from the SBOM pavilion, which served as a home base for SRA and was often abuzz with activity. Conference attendees dropped by for demos and questions while INL representatives made their rounds for submission reviews. Teammates stepped out to catch conference talks and returned excited to share what they had learned. Old friends, colleagues, and clients visited. It was hectic, it was fun, and most importantly it was eye-opening.
The SRA team mingling with S4x23 attendees in our SBOM Pavillion space.
Outcome
As stated in our previous blog post, “Developing SBOM as a Capability,” SRA entered this challenge with a differentiating goal. Our relationship to SBOMs is one where they are leveraged as a capability to enhance essential security processes like vulnerability management, asset inventory, or supply-chain risk management.
We have chosen to build this capability by extending an open-source tool to generate SBOMs, find vulnerabilities for components, and process VEX data. The star of our show was the FACT tool discussed in Part 1.
In this challenge, our team was able to demonstrate this tool’s extensibility by adding Windows MSI support on the fly with a simple unpacker plugin and a few Yara rules, through an enhancement allowing the export of CycloneDX SBOM files, and another adding support for VEX file ingestion. We learned quite a bit about the two standards but will save the philosophy for a follow-up blog post!
With our goal of an open-source submission and a capability mindset, we showcased the capabilities of open-source software and a highly innovative team.
SRA is tool agnostic and has always focused on helping our clients find the appropriate tool for their specific use case. While we’ll continue to use FACT and other open-source tools in combination with manual techniques when appropriate, we also recognize the value in employing commercial tools when deeper detections and a broader feature set is required by more mature organizations.
In fact, we hope to work together with any customer of commercial SBOM products and build a repeatable process for programmatically adopting and handling SBOM’s when they’re received or being given.
Open-source solutions like FACT can be ideal for ad-hoc, budget-constrained, targeted searches (e.g. by quickly adding custom detection logic for the next log4j emergency) or organizations just wanting to kick the SBOM tire before going all in. Overall, we find open-source software composition analysis (SCA) tools in this space as viable enhancements to existing or new processes when the cost or complexity of a commercial solution may not be justifiable.
Review of the Process
Firstly, SRA would like to recognize Idaho National Laboratory for agreeing to organize and oversee the SBOM challenge. They had the uniquely challenging charge of comparing multiple solutions with varying feature sets in a relatively new product category and gracefully managed what were at times, competing interests.
Although the challenge had some targets move throughout the planning process, due in part to the difficulty in obtaining representative firmware and sample files, INL still found a way to put important products in full display for the s4x23 target audience and to generally raise awareness amongst practitioners looking for a way to deal with the long-standing blind spot surrounding OT software and firmware.
As all solutions demonstrated the ability to export machine readable SBOMs, focus turned to the ability of each solution to perform SCA. This focus became clear during INL’s SBOM Challenge Debrief when three metrics were presented for each solution:
- Total components identified.
- Total files identified.
- Total vulnerabilities identified.
As an aside, while we weren’t surprised to see the commercial solutions outperform FACT with respect to detections, the variability between the number of detected components amongst the commercial products certainly stands out. Did some products fail to detect some components and vulnerabilities? Were others overly sensitive?
We believe and understand that the INL’s decision to present a quantitative summary was due to time constraints, as it would have taken an order of magnitude more effort to create a reference (answer sheet) and evaluate each SBOM in depth. However, it may have been beneficial to also find time, or a way, to focus on the accuracy of the SBOM contents, as that directly impacts vulnerability management and supply chain awareness.
Distilling the challenge down to the quantity of total vulnerabilities identified presented another issue – none of the reported vulnerabilities were validated. Any asset owner suddenly being dealt with a considerable overhead of new potential vulnerabilities could easily lead to overworked teams chasing false positives and missing real, valid, operational security threats.
The ideal solution for SBOMs will require both personnel and tools capable of careful discrimination to quickly wade through this potential noise to identify the vulnerabilities that present the highest risk to their particular organization and operation. SRA helps organizations develop their capabilities to leverage the risk ranking and prioritization features showcased by some of the commercial solutions to enable practical, right-sized steps toward improving security posture.
The final piece of the challenge was to demonstrate the ability to ingest VEX documents and display our analysis of them. We found this part of the challenge a bit underwhelming and thought that the simple “can you ingest VEX?” question did not spark enough discussion around a standard that holds a large potential value. Many of our conversations with attendees around VEX saw a ton of interest and led us to believe that there is large interest in the standard for vulnerability management.
Building a Better Bake-off
Ok, so the challenge succeeded in introducing the spectrum of solutions, check. But it stopped short of providing samples, reference data, and scoring that we believe would have allowed for a more apples-to-apples comparison of the critical features common to all these tools. This got us wondering how we might design a follow-up challenge.
SRA is no stranger to vendor/product bake-offs. We often perform them in client environments, in real time, with real-world data and test cases. Our goal is often to measure qualitative, not quantitative results.
If we were to perform a bake-off for a client investigating which SCA tool/feature is right for their vulnerability management process, we would start with building out test cases.
This would involve taking pieces of firmware from a client’s environment and applying subject matter expertise to identify software components and vulnerabilities through a multi-pronged process of reverse-engineering, OSINT, device test, etc. We would then feed that firmware into each tool to compare outputs. This gives us the ability to test for accuracy and validate the numbers that each tool reports.
We would then compare how vulnerabilities are reported by each tool, with the main criteria being actionable results. For firmware test cases with previously known vulnerabilities, we would validate that the tools report them and see how they are rated based on risk and remediation priority. Any novel vulnerabilities reported would be validated through hands on testing or research and false positives would be noted.
Another set of test cases we would provide would be sample VEX documents containing those identified vulnerabilities of components in the provided firmware. This would be to accurately test how the VEX would be ingested and interpreted by each solution.
After running the test cases, SRA would investigate the interoperability of each tool such as adherence to standards and API support. We want products to work as components of our clients’ environments, not as stand-alone solutions.
An important note on the delivery of the above is that the test cases should be completely blind. On the day of the bake-off, no vendor should have prior knowledge of any test case before hand to prevent modification of their tool to support better looking results. However, we often appreciate vendors who are willing to modify during the bake-off to add support as it often demonstrates their willingness to adapt to future problems.
We see a bake-off as an unbiased comparison of each participating solution working towards the goal of determining which is the best fit for the client’s environment and their specific use cases. It is not a measure of which solution is better, but rather a measure of which solution is better for the environment it will be deployed in. SRA is always finding new ways to make this fit more accurate for our clients and our SBOM capabilities is another example of that.
SBOM and VEX: The Clear Winners
The interest in SBOM and, by extension, VEX was palpable at s4x23 judging by the sheer volume of visitors in the SBOM pavilion over the course of the week.
When we chose the open-source FACT tool to be part of our people and technology SBOM solution, we expected to only have a fraction of detections compared to commercial products with teams dedicated to improving detections and consistently updated databases of vulnerability information. We were happy to see that the investments made by the commercial products have paid off as they proved to be more effective in both the quantity of components detected and vulnerabilities flagged.
We left s4x23 with a really good sense of what the industry has to offer and were able to speak with hundreds of attendees about their own thoughts and requirements surrounding SBOMs and VEX, from both the producer, consumer, and dual-role standpoints. We learned of various states of maturity, capability, challenge, and hopes for the SBOM tool related market.
The industry professionals, and more importantly the target audience of these solutions, provided the insight for us to say that there are still some limitations with the capability of SBOM and VEX related tools in general, both open-source and commercial. However, there is promise – let’s hope it lives up to the hype!
We were pleasantly surprised by the turnout and interest in the topics presented. S4 and similar events raise awareness, spark discussion, highlight strengths, and reveal weaknesses. These outcomes are invaluable to the necessary growth of the young OT/ICS security industry.
SBOMs were the star of this show, and each vendor was able to highlight the strengths of their products and capabilities to those attendees most interested in expanding their vulnerability management programs. Like a big SBOM speed dating event, we hope that many participants connect with the products that meet their needs.
To everyone we spoke with and those who were not able to make it but are following along; thank you for the insight and fruitful discussions. For those who want to get a bit more philosophical, stick around for part 3 of this series where we’ll dive a bit deeper into the SBOM and VEX standards, the philosophy behind them and their future.
As for SRA, we’ll continue honing our SBOM capabilities and using them to help clients harness this buzz, convert it into meaningful vulnerability management practices, decide on fit for purpose tools, and integrate them over time into a well-designed process.
SRAs Team at the S4x23 SBOM Challenge Debrief Session
P.S. A correction in defense of open source
During their debrief, INL had somewhat comically pointed out that the comparison they had built was not apples-to-apples but rather pocketknives to chainsaws.
In that metaphor, FACT might represent the pocketknife, which is compact, readily available, low cost (actually, free), and easy to deploy. It’s certainly not the tool of choice to bring down a tree, yet there are plenty of applications where it can be successfully applied to perform basic SCA tasks. Any missing functionality can be added through FACT’s well-documented plug-in architecture.
The challenge results presented on the final day of the conference are certainly not the complete representation of the capabilities added to FACT and of the native functionality available – there are many powerful plug-ins that were not necessary for the challenge and, as such, had not been enabled.
A slide displayed during the debrief indicated that FACT identified no files for each of the 3 challenge artifacts. In reality, FACT actually identified 291 files in the MSI, 1 file in the “baremetal” firmware, and 7836 files in OpenWRT, as reported in its web-based user interface. However, the enhancement that we made to enable SBOM support for FACT simply did not include individual file support in the SBOM files, though our development team indicated that this would be a “trivial” feature to add.
We invite anyone who is curious to learn more about FACT here or try it for themselves.
Gabe Siftar
Gabe specializes in securing IoT solutions, cloud environments, and hardware. Gabe has strong experience working with clients managing large deployments of IoT, medical devices, and Industrial Control Systems.
Prior to SRA, Gabe worked as a Systems Engineer for a global medical device manufacturer. This experience provides a deep pool of knowledge available to clients who design, manufacture, and manage secure hardware and IoT solutions. Gabe worked at the intersection of medical hardware, IoT and hospital IT systems, where he facilitated the development, deployment, and maintenance of critical healthcare systems.
At SRA, Gabe has applied that breadth of experience to client security assessments and as an advisor for client projects that span from on-premise to the cloud, and from low level hardware security to system architecture.
Walker Hines
Walker focuses on long-term red team operations, external and internal penetration testing, and spearphishing. Walker is a member of SRA’s R&I team, which researches and develops tools, techniques, and public content.
Walker has worked for clients in several industries including banking, pharmaceutical, insurance, entertainment & media, healthcare, manufacturing and retail.
Walker’s background includes Windows, Linux, System Development, and Software Development. Walker is a graduate of the University of Rochester's Computer Science program. He also is a Certified Red Team Operator (CRTO).
Jason Rivera
Jason specializes in Cyber Physical Systems (CPS) security strategy program development and controls. He has spoken at H-ISAC, ISACA and multiple CPS security trainings while also serving in OT/IoT security leadership roles since 2018.
He has prior experience with Cybersecurity Operations, Incident Response (IR), Network Security and Risk Management.
In addition to helping clients, Jason has been a go to for SRA in new and emerging services. He has also been a Palo Alto Networks Certified Network Security Administrator (PCNSA) and Cisco Certified Network Associate (CCNA) certified.
You must be logged in to post a comment.