This blog is part of a blog series detailing best practices for operational technology (OT) cybersecurity for under-resourced organizations by Dragos OT-CERT (Operational Technology – Cyber Emergency Readiness Team), which provides free resources to help small and medium businesses (SMBs) create or enhance their OT cybersecurity program. Not yet a member? – join OT-CERT and get started today.
Larger Organizations Take Note
If you have been increasing your security posture and reduced risk of a significant cyber attack in your enterprise, including your OT environment, that’s excellent news! However, does your risk assessment include the possibility of a cyber attack on one of your critical suppliers, and the impact that would have on your company’s operations? Could you still produce your product or provide services to your customers? Read on to ensure that you are quantifying the likelihood and impact of that risk correctly in light of the current threat environment. And strengthen your supply chain security risk posture by promoting OT-CERT to your suppliers.
Legal Disclaimer
OT-CERT resources are intended to provide guidance to help under-resourced organizations, those lacking sufficient financial resources or technical expertise, to establish minimum baseline OT cybersecurity protections and do not necessarily meet the usual best practice standards for a mature OT cybersecurity program. Dragos, Inc. does not provide any warranty or guarantee that following the guidance provided by OT-CERT alone will safeguard an organization from all OT cybersecurity threats. Whenever possible organizations should seek additional enhancements to the recommendations provided by OT-CERT resources based on an organization’s own cybersecurity risk profile.
How to Disconnect Your IT, Demilitarized Zone (DMZ), and OT from Each Other
“Gotta keep ‘em separated” – The Offspring
Even though they’re an 80s and 90s punk rock band, The Offspring had the right idea for OT: gotta keep the networks separated. However, as music evolved, so did the need for more visualizations in the OT networks. Now I can listen to any song I want on YouTube and check in on a power plant, all at 30,000 feet in an airplane. However, if you’re not careful it could be extremely easy for both you and your adversary to work remotely. If an employee can remotely access your OT environment or there is a data path between IT and OT, then an adversary could also gain access to your environment if it’s not secured correctly.
During our consulting engagements, we encounter many customers who do not consider an incident response scenario in which they might need to disconnect their IT, demilitarized zone (DMZ), and OT from each other.
Consider this hypothetical situation: You receive an alert from your endpoint detection and response (EDR) solution that a corporate workstation has gone offline, and it turns out it’s been ransomware’d. You receive more alerts; the ransomware is worming its way through the environment, and it’s starting to go after servers. So far, your Production (OT) network is humming along, and no one’s reporting any issues, but you know if it gets into the OT network recovery will be much more difficult, take much longer, and have a much bigger financial and reputational impact on the company than ransomware in IT. Furthermore, you know that many cyber attacks impacting OT begin in IT and traverse into OT – (insert more panic).
At this moment, you ask: What can one do to quickly eliminate the threat from the OT network?
The answer: Disconnect.
But you need to keep in mind the potential ramifications of doing this. Colonial Pipeline did just that – disconnected their OT from their IT. As a result, they saved their OT network from the ransomware but also left the company unable to bill their customers. This is a tricky situation and is a lot more complicated than it sounds so you need to plan ahead.
The goal is to remove ANY chance of IT traffic crossing into OT. Three of the most common options we at Dragos see or recommend are pulling cables, shutting down network interfaces, or powering off network gear.
Questions to Consider BEFORE You Disconnect
So let’s rewind a little because hopefully you’ve planned for this… and you know:
- How and where to safely disconnect – which cables can be pulled and in what order to avoid chaos.
- What mission-critical or business-critical information is being transmitted to the IT network? Billing, environmental readings, regulatory historical data? If you’re a manufacturer, can you receive work orders and create final products?
- How long can you remain disconnected before you must stop operations due to safety concerns, product backlog, and regulatory requirements?
- Can you operate manually, or do your robots need input from an enterprise resource planning (ERP) system to engrave serial numbers?
- If you sever the network connection, can you still communicate with remote sites, or does your remote traffic traverse IT equipment?
Key Points to Consider in Creating a Disconnect Plan
Don’t know the answer to those questions? Below are key points to keep in mind when creating a disconnect plan:
- Assessment of the Network: How is the network configured and what would be “natural” choke points to use. e.g., if all the OT/IT traffic is going through a DMZ firewall, that would be the first point to consider. The goal is to understand how to disconnect totally/partially OT from IT.
- Assessment of the Threat: Understand the nature and extent of the threat. Determine which systems could be affected and how the virus or ransomware would spread from IT into OT.
- Documentation of the Plan: Plan/document what is required for isolation before anything else. Do I need credentials to access management interfaces for the equipment? Do I need physical keys to enter the rooms where the equipment is installed? Is there a specific consideration before unplugging or shutting down equipment?
- Documentation of Reconnect Process: You need to know how to put everything back to normal and this should be tested as well. This could include changing passwords, order for re-connection, etc.
- Communication Plan: Develop a clear communication strategy to inform all stakeholders, including IT staff, management, and users, about the planned disconnection. This helps in managing expectations and coordinating efforts.
- Get Executive Approval: Make sure you communicate the risk to your executives before an emergency happens and get their explicit approval to take action. They need to fully understand the ramifications of disconnecting – remind them of Colonial Pipeline! Do they require that you get their approval during an incident before taking action or are you empowered to make the decision in time of crisis? An executive tabletop exercise can be an effective tool for doing this.
- Backup and Data Preservation: Ensure that all critical data is backed up before disconnecting the network. This helps in preventing data loss and facilitates recovery efforts.
- Isolation Strategy: Plan how to isolate the infected network segments from the rest of the network. This might involve physically disconnecting cables, disabling wireless connections, or using network segmentation techniques.
- Testing and Simulation: Conduct tests and simulations to ensure that the disconnection plan works as intended. This helps identify potential issues and refine the plan. This can be tricky – be very careful that testing does not impact production!
- Resource Allocation: Ensure that you have the necessary resources, including personnel and tools, to execute the disconnection plan effectively. This includes having antivirus and anti-malware tools ready for use.
- Post-Disconnection Monitoring: After disconnecting the network, continuously monitor the situation to ensure that the threat is contained and that no further infections occur.
- Post-incident Documentation: Document the entire process, including the steps taken, the rationale behind decisions, any issues encountered, and lessons learned. This documentation can be valuable for post-incident analysis and future reference.
After reading this blog you might feel like this is too complicated and too much trouble to worry about, but it’s important for you to understand and document this risk and prioritize mitigation as it makes sense for you in your overall OT cybersecurity strategy. Unfortunately, ransomware threats continue to escalate substantially – especially in manufacturing. We read about companies whose operations are impacted by ransomware in multiple ways: some suffer significant downtime when ransomware brings down operations. Others bring operations down proactively to be safe – recovery is easier but there is still revenue loss. If you have time to figure out a disconnect process it could reduce downtime risk as well as financial and reputational impacts.
Stay Up to Date with OT-CERT Resources
Dragos OT-CERT offers FREE resources to help SMBs build their own manufacturing / OT / industrial control systems (ICS) cybersecurity program without hiring any cybersecurity experts. OT-CERT membership is free and globally available to OT asset owners and operators. Resources are oriented toward small and medium businesses and resource-challenged organizations with OT environments that lack in-house security expertise. Members have access to a growing library of resources such as reports, webinars, training, best practices blogs, assessments, toolkits, tabletop exercises, and more.
Currently available resources include:
- OT Cybersecurity Fundamentals Self-Assessment Survey
- OT Asset Management Toolkit
- Self-Service OT Ransomware Tabletop Exercise Toolkit
- Collection Management Framework for Incident Response
- OT Cybersecurity Incident Response Toolkit
- OT Data Backups Guidance
- Host-Based Logging and Centralized Logging Toolkits
- Secure Remote Access Toolkit
- Network Segmentation Toolkit
- Firewall Configuration Toolkit
- System Hardening Toolkit
- Default Passwords & Internet-Exposed Devices Toolkit
- Change Management Toolkit
- Risk-Based Vulnerability Management resources
- Access to introductory ICS/OT cybersecurity courses in Dragos Academy
If you haven’t joined Dragos OT-CERT don’t delay! Membership is open to organizations that own or operate a manufacturing / ICS / OT environment. Please join and spread the word to your community and supply chain so we can all work together to raise the security posture of the entire ecosystem – we are only as strong as our weakest link.
We look forward to working with you to safeguard civilization!
Join OT-CERT today!
Ready to put your insights into action?
Take the next steps and contact our team today.