Service and platform providers in the pen testing, red teaming, and attack surface management spaces are touting the benefits of “continuous” testing. This is actually a good idea, but whether you insource or co-source it, continuous should not mean single-threaded. There are different testing activities that make up a robust continuous testing program. A continuous testing program comprised only of red teaming or pen testing, and an attack surface management program focused on a tool’s single pane of glass are narrow views of what security teams should cover and resource. Continuous Testing Programs, like great chocolate cake, have layers, great ingredients, and avoid too many cooks pastry chefs in the kitchen.
Why? The Narrative of the Continuous Testing Program
When I have a mouthful of chocolate cake, feel-good endorphins shoot from my brain to my toes. I’m not sure the feeling is supposed to go that far down, and I should probably talk to a doctor. CISOs know that they can give Senior Management, Internal Audit, and regulators that same head-to-toe feeling by touting their Continuous Testing Program. It sounds good. It looks good on a strategy slide. It’s congruent with the idea of 24×7 SecOps which has become table stakes. The narrative of a continuous testing program, whether you go big or small with it, is a no-brainer. Regardless of the size of the program, any cake is better than no cake. Large or small, a great confection is made by following a recipe, using great ingredients and layers.
Layers: Vulnerability + Visibility
Many continuous testing programs miss the distinction between two separate objectives:
- Vulnerability is understanding exploitability at a point in time.
- Visibility is acknowledging that the organization can and will be vulnerable (often through no wrongdoing of its own). Visibility focuses on detecting when a vulnerability has been exploited so that the security team can respond.
Many continuous testing programs are focused on identifying and closing vulnerabilities on a streaming basis through a combination of automation and manual testing. This is a good start, but those programs miss half the equation. Visibility is the more difficult half to manage because there are fewer mature processes and qualified people to support it. If you’re still thinking about cake we would probably get along, and yes, Vulnerability and Visibility are the two essential layers of your continuous testing program.
Vulnerability Ingredients
Vulnerability Management. You are already doing one of the process-oriented things in the recipe! It’s not sexy, not new, and it relies on automated scanning. But it’s far from useless because it identifies low-hanging fruit at scale. If you need a narrative boost and have enough attack surface, this is a testing process that can be described as continuous in nature.
OSINT. Some organizations’ attack surface management platforms and processes include notification of Open-Source Intelligence (OSINT) hits. These are usually cleartext passwords, encryption keys, and system information in publicly accessible resources like GitHub and many other lesser-known coding and sysadmin platforms. Perhaps the subject of another future post, suffice to say for now that if your organization outsources application development this should be essential for you. The SaaS nature of many of these services can also contribute to your continuous narrative.
Pen Testing. One of my clients told me recently they’re under pressure to do more than one external penetration test per year. They are in a low regulation industry, but this was an expectation of one of their partners. I replied that other industries often do more than one for good practice, but most aren’t required to do so. I added that showing their partner a filled-out calendar of more varied testing activities can likely give them the comfort that testing for security vulnerabilities is not just a point in time for them but part of a process including their vulnerability management and other activities we discussed (and which I list below).
Another client was pitched by a service provider with a “continuous pen test service” that appeared to be a series of pen test sprints throughout the year (not 24×7, nor should it need to be). I suggested they would reach diminishing returns for the budget spent and should look to do a healthy dose of pen testing but not spend all their continuous testing budget on it. Why? Pen testing is simply not the best tool for assessing and improving Visibility. Please allow me to explain.
Visibility Ingredients
Purple Teams / Adversary Simulations. Pen testing is good at identifying a vulnerable and exploitable state, but it barely tests visibility. Pen testing only cuts a narrow slice of the content of MITRE ATT&CK® which is our best map to test and improve visibility. Cross-functional security teams who conduct side-by-side, collaborative Purple Teams / attack simulations do the most to test whether a larger set of priority detections will fire at the right level of severity, with the right level of information for SecOps to respond. Visibility improved by Purple Teams inspects what it expects. This means we do drills to practice visibility until we get the detection outcomes we need and increase the chance that SecOps will prevail in a live fire fight.
Attack Surface Monitoring (ASM). ASM has been co-opted by vendors and can vary in meaning. If you have a large attack surface (thousands of external systems and tens of thousands of internal systems), having a platform that gives you high fidelity alerts on new data center and cloud systems/services can be a great way to inform your program of changing visibility and some vulnerability conditions. The automation and near-real-time monitoring also loosely qualifies as continuous.
Cautions
Too Many Pastry Chefs in the Kitchen
It is important to design and follow a recipe for your program based on your needs and budget, and it’s the CISO’s job to present and sell the completeness of the program. Watch out for these rogue would-be pastry chefs who want to add too much of an ingredient, or something you don’t need at all:
- Security platform vendors, who insist their AI and automation means you don’t need smart analyst time to validate vulnerabilities or visibility gaps. Quality security has never been fully automated. Testing tools create a lot of noise and too much data is unmanageable. Start with effective process, reasonable scope, and grow the program over time if needed.
- Service providers want to sell you the service they’re good at and what they have in supply. You don’t need year-round of whatever they do just because that’s what they offer. Service providers should not have an undue influence on your requirements.
- Board Members and Audit Executives, many of whom are obsessed with Red Teams because that’s what they know and understand. In my experience most executives who insist on a red team don’t know the difference from a pen test. (And you should be pen testing and especially purple teaming more than red teaming!) Red teams are very drawn out and if done correctly avoid every system and control that you’ve put in place. They are a good test for a mature organization. If you haven’t practiced through a calendar of purple teams and pen tests first, you haven’t even studied for the red team exam. Why would you pass?
Frequency and Reliance on Breach and Attack Simulation Tools (BAS)
Continuous testing on steroids is daily BAS testing. Like using steroids, this type of testing can look impressive but have negative consequences. BAS sales are targeting our industry with a loud but thin narrative that testing systems for visibility every day through BAS is a robust, mature practice. I disagree. BAS is another security system that needs to be implemented and usually involves agents in the network. It requires tuning and care. It’s a cross-team effort that takes months of focus and a significant cost. So, it better be worth it, but:
- BAS creates a lot of repetitive data and can quickly lead to “so what?” It sounds good to say we’re testing every day, but are we even looking at the results? Malicious behavior starts to look normal on the network and SecOps gets numb to it. Systems with BAS agents will be ignored by SecOps, and if I’m a red teamer or a threat actor, one of those systems is a presumed innocent staging ground for my attacks!
- What’s the benefit of a subset of your systems constantly running attacks on themselves? Unlike vulnerability management whose ambition is to scan every system, BAS covers a sample. (Note: this is also a limitation of the team-based purple team approach I advocate). We will not be able to test exhaustively for visibility across the whole network, but should we even want to? Red Teams and Pen Testing have never done that either. It’s a greater benefit to run less frequent collaborative purple teams so stakeholders can practice detection of anomalies. Volumes of false positives from BAS are unlikely to improve visibility of real attacks coming from non-BAS’ed systems.
- BAS can help identify a downed log source, but your SIEM can already do that.
BAS is the elaborate fondant frosting on top of the cake. It looks appealing in the bakery window but it neither tastes great nor is it worth the calories.
Last and Most Important Ingredient for Success: Resource Allocation
Your continuous testing program needs tools, some mentioned above, but it also needs people. If you can insource it, make sure your team is working collaboratively, achieving the calendar of activities, and planning for their replacements. The greatest downside to insourcing is, well, counting on your talent to stay and operate the program in our current security jobs climate.
If you want to co-source this function, I recommend in-sourcing your team leadership and remediation resources, which will help you consistently manage the success of the program, and co-sourcing the technical talent for pen testing and purple teaming. Service providers with wide capabilities and deep benches can bring different perspectives over time and swap activities across calendar years per your changing needs. Service providers should be willing to help train junior members of your team so you have the option of building internal capabilities when you have the vision and backing for a larger internal team on the horizon. If you do ask a service provider to help you train junior talent, have tempered expectations for how long it will take for your new team members to become proficient.
Tim Wainwright
Tim has been a speaker at RSA, Gartner, FS-ISAC, H-ISAC and (ISC)2 National Congress. Tim helped found Security Risk Advisors in 2010. Tim advises CISO Offices on modernizing cybersecurity strategy to improve governance, communication, team culture and growth, detection and response capabilities. Tim is a thought leader in the area of purple teams and attack simulation and metrics to describe quantified threat resilience. Tim has a background in penetration testing, security assessment, and frameworks.
You must be logged in to post a comment.