In budget-tightening times, a security data pipeline can greatly reduce your SIEM costs and increase log efficiency. SRA has implemented this new approach for many of our clients, helping them realize significant savings by reducing log size and volume ingested by their SIEM platforms. This approach applies to most organizations unless you are locked into a long-term, fixed cost SIEM contract, don’t like to save money, or have figured out the secret to obtaining unlimited budget. If the latter applies to you, the CISO in me would like to know your secret.
Think About Logs Differently
Myth 1: Logging EVERYTHING you can to SIEM will result in better detection.
Truth: Not all logs are created equally. Some logs are key for alerting and result in actionable events for SOC analysts. However, most logs we send to our SIEMs are not used to generate alerts. Instead they are useful for investigations, SOAR enrichment and threat hunting. Those logs do not belong in the SIEM!
Myth 2: Compliance requires all your logs to be in the SIEM.
Truths: Compliance-related logs need to be accessible and searchable. This can be accomplished with much cheaper security data lake storage. SIEM costs continue to increase, especially when leveraging a cloud-based platform. SIEM vendors capitalize on the “log everything approach” and a SIEM is very expensive when compared to the cost per GB for data lakes.
What is a security data pipeline?
A traditional logging infrastructure sends the full log volume from our sources directly to the SIEM. You may have a scheduled job that purges data or sends it to an S3 bucket to reduce retention volume within your SIEM, but you are still sending everything to your SIEM and paying for a high daily ingest rate.
This often forces one of two, bad but avoidable decisions: do I increase my SIEM license to ingest more data, or do I selectively choose which logs I can’t afford to ingest? You don’t have to make that choice!
The first graphic below depicts a traditional logging infrastructure, while the second represents an optimized security data pipeline.
On the left: Traditional Pipeline; On the right: Optimized Pipeline
A security data pipeline uses logging middleware that 1) reduces the size of log files by eliminating unnecessary or redundant fields and 2) intelligently routes only the needed logs to a SIEM, with the rest going to a security data lake (or both). It is possible to focus only on log size reduction, but the greatest cost savings comes from redirecting high volume logs such as firewall allows and drops, netflow, URL filtering logs, DNS, and DHCP data to lower cost security data lake storage.
Cribl Stream is a product that SRA has found to work well for this purpose. There are others, but Cribl provides scalability and enterprise-level features that you need for such a critical component of your logging strategy. With this architecture, you replace dedicated log collection/forwarding infrastructure such as syslog servers and intermediate forwarders with Cribl Stream. Cribl serves as a pass-through mechanism where it applies configurations to send data required for detections to your SIEM after reducing unwanted fields within log files and everything else to your data lake.
In a separate blog post we will provide additional details around Cribl configurations designed to reduce log file size and specific events you should send to a data lake. In the meantime, below are two case studies to highlight some of the cost savings we’ve helped our clients achieve.
Fortune 200 Company
Enabled ~$500K in annual savings |
Healthcare Provider
Realized ~$740K in annual savings |
As a service provider who offers 24×7 CyberSOC services, performs SIEM engineering and content development, as well as world class Purple Teams, we know what sources and alerts are necessary for our clients to detect both common and advanced attacker TTPs. Combine that with our ability to architect an effective security data pipeline using Cribl Stream leveraging our library of Cribl configurations to cleanse and route data sources, we are in a unique position to help our clients reduce their SIEM costs without sacrificing alert fidelity.
Michael Polise
Mike specializes in program strategy and execution for Blue Team controls. He has a broad focus across capabilities such as logging and monitoring, data protection, endpoint, perimeter, cloud, and data security with an emphasis on architecture and engineering.
Mike has extensive experience encompassing a wide range of technical and procedural controls, including program development. His experience allows him to focus on long-term strategic goals by identifying cross-platform synergies and developing comprehensive, effective approaches for defending against today’s security threats.