TL;DR – The best way to get started with Cyber Physical System Defense is to create a defense plan, collect asset inventory, and begin implementing monitoring infrastructure. These steps are all much easier said than done.
Let’s say you just found out that your organization has opened a new facility to support manufacturing demand for a new product. The board is concerned about the security of this new facility, and you’ve been asked to define and implement a strategy to defend this new infrastructure that you didn’t even know you had 5 minutes ago. As Cyber Physical Systems (CPS) security starts to become more prominent, these types of requests are becoming more common. How does someone deal with this?
We work with clients across the world to navigate these types of requests and have compiled a set of vendor-agnostic guidelines to assist with securing their newly found CPS environment. We’ve previously discussed how to approach “What is OT?” a basic question with a complex answer, that you need to understand before attempting to secure CPS environments. Feel free to check out that post to get an idea of the types of devices that we are working to secure in the CPS realm.
Planning a CPS Security Defense Strategy
When creating a new CPS security defense strategy, significant planning needs to be involved. Here are some best practices:
Socializing the CPS security strategy to all of the teams that may be affected/involved is critical to a CPS security defense plan. It’s important to let all the teams know what is needed from them from a time investment perspective, because nobody likes to have their time wasted.
SRA has helped guide our clients through these conversations in the past, and we’ve learned that the last thing you want to happen is a team being caught off guard in the middle of a tool implementation or site assessment. The team that’s caught off guard will often try to slow down or stop the implementation, and that’s not good for the productivity of your team, their team, or your budget! This underscores the importance of communication before the project starts.
Starting your CPS defense strategy with Asset Inventory
This is where most vendors would tell you to buy their product, but unfortunately building a worthwhile CPS defense starts with a process that is much less exciting: asset inventory! One of the biggest obstacles in CPS security defense and monitoring is not knowing what systems are out there, so having an asset inventory is critical. Obviously, tactics for starting or improving your asset inventory depend on how much budget you have available, but you can have a good asset inventory without spending a ton of money. Here are a couple of approaches you can take to catalog your CPS assets:
1. Centralize what you already have (Lowest budget required)
If you’re anything like many of SRA’s manufacturing clients, you have decades of inventory records maintained across several systems. Often these inventories are outdated or incomplete. The team that manages your CPS assets probably already has a way of cataloging and keeping track of them. You should talk with those folks to understand their way of doing things and potentially request/extend access to your enterprise security team.
You should also look to push your team and the site operations team that manages the assets to pick a source of truth where everyone publishes updates. This should be based on whatever is easiest for your operations team as they will require using the inventory to do their jobs more often than the security team. Centralization is all about picking a plan and sticking to it, no exceptions.
2. Select a monitoring tool to be the source of truth
This is probably the most expensive route and can probably only be considered if you have a larger budget. Why is this so expensive? Because selecting a monitoring tool will probably be a time-consuming process as you should consider different tools with different feature sets, and you’ll also want input from the operations folks on the shop floor along with the security team. In addition, you’ll need to spend additional capital on deploying the tool, which will require some physical installation expenses (like shipping, cables, labor, etc.).
No one team or group should unilaterally decide on a monitoring tool without the other. What we mean by this is the security team (or even the CISO alone in some cases) should NOT be buying a tool without talking to his operations counterparts. If all teams are engaged and informed during the purchase, then the enthusiasm and buy-in for the tool will be much higher across the organization.
More tips and tricks for selecting a monitoring tool under step 4 below, but the moral of the story here is that most CPS monitoring tools have a cataloging capability that can be leveraged for your asset inventory, but you need to have a plan to implement.
3. Add CPS assets to your enterprise CMDB
The most obvious solution could end up being the correct one for your team, updating your current enterprise CMDB to include CPS assets. This might seem like the obvious solution for your security team, but you should take your CPS operations teams’ opinions into account here as well as you need to make sure that they have the access the CMDB, and the ability to store all the necessary data that they need to do their jobs or match how they are currently cataloging assets.
Even a basic set of fields like hostname, operating system, IP, location, and vendor name will be enough to greatly enhance your defenders’ capabilities. This will be a very large effort, but you should have some previous projects that can guide you on how to add these new assets.
It’s important to note that sometimes CPS assets shouldn’t be in an enterprise CMDB. Sounds crazy to say, but there are drawbacks to having everything in the enterprise CMDB like when IT compliance tries to launch compliance objectives against CPS assets that are unprepared to pass or in some cases not possible to pass. This option should only be considered if you can avoid CPS/OT assets causing disruptions to an existing CMDB.
What are our options for CPS security monitoring?
If you select Option #2 above, you’ll need to purchase a new tool. Before you buy a new monitoring tool for your CPS environment, ask your team and the operations teams the following questions:
- Where are our largest blind spots for security monitoring today?
- What OS(s) do the system(s) you are trying to monitor have? What is your process for updating/securing those OS(s)?
- What expertise, if any, does our CSOC team have when it comes to monitoring and escalating alerts coming from CPS assets?
The graphic below gives sample answers to the questions that you would ask the security team. There are other factors in choosing a monitoring tool, but if your answer matches one of the cells in the graphic this should help you decide which option to further explore. We also break down the options further later in this post.
Knowing the answers to these questions will help push you in different directions when it comes to setting up a new monitoring tool in your CPS environment, and it’s important to remember that these assets are often the business’s most important assets when it comes to revenue generation. Security needs to find a way to monitor suspicious activity on these assets while not causing impact to any CPS process. You may need to mix and match options below to fit your team’s needs:
Option #1: CPS-specific security monitoring tools
This is the most common way a security team will begin CPS security monitoring. We’re not here to make recommendations on which vendor is best. It will depend on your team as well as what your primary use cases are. Some of the top vendors in this space are: Claroty, Nozomi, Armis, and Microsoft Defender for IoT, and each of them have different areas that they are better at detecting. This functionality may also be available in other products like the “IoT security” blade in Palo Alto’s. Talk to your existing vendors and see what offerings they have for OT/ICS/IoT.
Going back to the answers to the questions above, these OT monitoring tools are great whenever there are very large blind spots that you need to monitor that happen to have very “sensitive” equipment that you wouldn’t be able to install an agent on (i.e., factory floor PLCs, freezer equipment, building automation systems, etc.). These tools often act like Intrusion Detection Systems (IDSs), and this is great for trying to get visibility into new areas without “deploying” anything on the actual machines that you are trying to get visibility into. These tools can generate alerts based on network traffic. Some tuning will be required before forwarding alerts to a SIEM because there will be some unknown traffic patterns in these environments that have never been monitored by enterprise security before. In some cases, you’ll find completely different ports and protocols in use at CPS sites or environments, so you’ll need to tune these tools differently than your enterprise network monitoring tools.
Something else to consider when buying a new security monitoring tool is the expertise of your team. If your team does not have any experience working with CPS assets, or if your team has little to no IT operations experience, then you may have to rely on vendor provided training to help get everyone up to speed.
Keep in mind that maintaining communication lines with your teams that manage the CPS assets is critical before, during, and after the engagement. After is especially pertinent, so you can discuss recent alerts and how they want to be notified by your CSOC.
Option #2: Traditional security monitoring tools
If you have little to no budget available to you, this is the route you should take. There are plenty of options in this space, but it is very “old-school” security.
Tools like Snort, Zeek, and Suricata can be your friend when attempting something like this as they are free and plenty of guides and documentation exist to help you stand them up. It’s important to note if you go this route you need to have somewhere where you can store the alerts coming from them (like a SIEM or data lake). Otherwise, you are putting forth a lot of effort to create more alerts that don’t go anywhere.
Your team also needs to have strong competency in network security to really get value out of this as there will be lots of noise to sort through at first. You will also be better off correlating the alerts from your open-source tools with your firewall logs.
You may also consider using your existing enterprise-grade network monitoring tools, but these are often not the best solution because these tools are not OT-protocol aware. Also, traffic patterns may be different between enterprise and CPS environments, which will make tuning out noise difficult. Obviously, adding additional sensors of your enterprise tool to your CPS environment is better than having no visibility, but you’ll find that its not the most ideal solution.
If you don’t have a firewall, then I’m surprised you’ve read this far! You should probably start with installing a firewall in between your CPS environment(s) and your enterprise environment as a starting point to provide a base level of protection, and you should by no means prioritize monitoring if you don’t yet have inbound traffic monitoring/protection
Option #3: Active defense monitoring tools
Active defense means essentially whatever standard monitoring or protection tools (think AV/EDR) that are appropriate for your CPS environment but have active components, not just passive. This often isn’t an option for OT/ICS for device performance reasons and is another important topic that should be brought up during the planning phase. You may encounter systems that are essentially servers out of your monitoring purview, and in that case, you may want to explore just installing your enterprise server/desktop tools. Obviously, this won’t be the case for all devices critical to manufacturing processes, which is why talking to your site operations folks that manage the systems is once again critical to get an idea of when you can/cannot install various agents. A high level of testing must also be incorporated when taking this route to ensure no business interruptions will be caused by your agent deployments. You may also need to determine a plan for running agent updates and upgrades that align with already scheduled shutdown windows.
The largest benefit to choosing this option is for your CSOC team to not have to undergo any additional training or have any additional knowledge beyond what they already have. The only additional content that will need to be created is new communication workflows when alerts are generated for these systems. In some cases, you may also be limited in automated response actions. This is something that should also be discussed with the operations team that manages these new devices during the deployment process.
Conclusion
Any cyber defense conversation is going to have a budget talk. We’re not here now to discuss how much or how little you should be spending on your security in today’s world but, you can build a secure CPS security program on any level of budget and having a small budget shouldn’t lead to a strategy that ignores CPS security entirely. The more funding your CPS security program has, the better, but you can even start the steps listed in this post under the umbrella of enterprise business-as-usual security activity. CPS assets will often be the biggest “money-making” devices a company can have, so securing them is critical no matter how much money is available to spend.
It’s important to remember that these options are not all or nothing! Security teams can pick and choose how they want to defend their environments and we encourage everyone to start a dialog with the people that work in these areas every day. You’ve just read a lot of different and unique options for setting up a CPS security program above, but the end result is not that different from securing what your cybersecurity teams have always secured. The only difference is the conversations you will have along the way! You may be shocked to learn how security conscious the folks on the factory floor are, and your company will be more secure because of your efforts.
Feel free to reach out if you’d like to learn more on this topic. Look at some of the CPS security services we offer. This is what we do, we are experts and are happy to help you on your journey.
Zack is SRA’s subject matter expert on OT detection tools. He writes OT-specific detections for various detection tools across both the network and endpoint stack. He also has created logging architecture in multiple large corporate and OT environments in different industries.