This is our monthly blog detailing best practices for operational technology (OT) cybersecurity for under-resourced organizations by Dragos OT-CERT (Operational Technology – Cyber Emergency Readiness Team), which provides free resources to help small and medium businesses (SMBs) create or enhance their OT cybersecurity program. This month’s best practice recommendations cover the following categories from the OT-CERT OT Cybersecurity Fundamentals Self-Assessment Survey: Asset Inventory, Configuration Security, Cyber Risk Management, and Cybersecurity Incident Response. Hopefully, you filled out the survey and identified your gaps – these best practices can be implemented to begin to address those gaps. If not, there’s no time like the present – join OT-CERT and get started today.
Larger Organizations Take Note
If you have been increasing your security posture and reduced risk of a significant cyber attack in your enterprise, including your OT environment, that’s excellent news! However, does your risk assessment include the possibility of a cyber attack on one of your critical suppliers, and the impact that would have on your company’s operations? Could you still produce your product or provide services to your customers? Read on to ensure that you are quantifying the likelihood and impact of that risk correctly in light of the current threat environment. And strengthen your supply chain security risk posture by promoting OT-CERT to your suppliers!
How to Implement Change Management in Your OT Environment?
Change is inevitable, except from vending machines. – Robert C. Gallagher
And your ICS/OT (Industrial Control Systems / Operational Technology) environment is no different. Our last blog discussed why a change management process is important for your OT environment. In this blog, we recommend how you should implement a change management process for your OT environment. We’ll get into the weeds of what a basic change management tracker would look like and how to fill it out.
Below are examples of the attributes we recommend you track as part of your change management program. We’ve also included an example workflow of the change management process, who’s involved, and how the change management tracker is used throughout the entire process.
You can download a template of a change management tracker on the OT-CERT portal under Resources. Register today to get access!
Change Identifier– Unique Identifier Assigned to Change
This is the unique identifier for a change. This can be numbers, alphanumeric, or anything else. The purpose of this is to easily identify a change utilizing a unique identifier, like a Primary Key in a SQL database. Keep in mind that it is possible to have several changes logged on the same date, changes that only affect half of the servers, and different software requirements on different machines, for example. In the example below, we used the year the change was opened and then a sequential order. You could also use YearMonthDate and sequential number. (Eg. 2023APR3210001.) The only limit is from Excel at 32,767 characters per cell 😁.
Change Manager – Person Responsible for Change Success
The Change Manager will be person responsible for the change, from start to completion. This person doesn’t necessarily have to do the technical work – that is the Change Team – just act as a liaison. Or it could be the same person who completes the entire process, including test/production implementation. Some technical folks aren’t the best at communicating or writing things down, so the Change Manager should be a decent communicator and reliable with paperwork. This should always be an identified person, preferably a full-time employee, so there is someone responsible for ensuring the change is complete or as a point of contact for questions/issues. Other team members can be added as they take action during the change cycle, but need to be identified by their role, such as tester, implementer, quality assurance, or evidence collection.
Hardware/Software/Script Change Description – High-Level Description
This section will be used to identify what will be implemented. This section should be as descriptive as necessary but not too descriptive as to take up too much time reading or induce confusion. As discussed in the why change management blog, system owners, stakeholders, and other users will use this information to determine if a change might have affected the environment. It will also help, at a glance, to show what’s happening in the environment such as software installed or assets that require too much maintenance compared to the cost of the asset.
Date of Change Notification – Date Stakeholders Were Notified of Upcoming Change
The change could be initiated due to upgrading a system, vendor patch release, general improvements, or many other scenarios. This section provides a record for the Change Manager to show that System Owners and Stakeholders have been notified of an upcoming change to the environments. This notification will be in the form of an e-mail or other trackable and auditable communication method and will include as much detail as possible about the change, including affected systems, technical details of the change, why it’s being implemented, and recommended dates.
System Owners and Stakeholders – Leadership Involved in the Change
These are the people who need to be notified, give permission, and be provided with results of implementation in test/production. These people could be shift leads, managers, CISO, CIO, customers, and / or vendors. The Change Manager along with System Owners and Stakeholders will discuss how the change will affect the infrastructure, test pass/fail scenarios, and whom to contact from various groups for change completion verification.
Pass/Fail of Testing – Results of Testing
After implementing the change in Test, did everything work as planned? To determine pass/fail, the Change Manager and Change Team will need to follow testing criteria that was agreed upon with System Owners and Stakeholders. If the testing failed, the policy at the company and the criticality of the change will determine if further troubleshooting is required with the vendor or the change should be postponed until a new change identifier is initiated. In the example below, the second change failed in testing and therefore the remainder of that row will be blank in the sections that follow.
Date of Production Approval – Date that System Owners and Stakeholders Approved for Implementation
This is the big day. Change software/hardware/instructions are at the ready, operators and engineers have been notified if/when the change will be implemented, and everyone is on higher alert for issues and potentially switching to manual operations if the change goes awry. This date is agreed upon between the Change Manager and System Owners and Stakeholders. As discussed in the previous blog, not everyone knows what’s going on at all times, so having agreed upon times/date helps to ensure a smooth implementation.
Pass/Fail of Production – Results of Implementation
After implementing the change in production, did everything work as planned? This is where you communicate with concerned parties to make sure the environment is operating as designed. Operators, admins, and engineers ensure that what was implemented is working and identify if it is negatively impacting anything else. As discussed in Pass/Fail of Testing, the Change Manager and/or the Change Team will have worked out a testing plan in production that satisfies concerns of the System Owners and Stakeholders. In the example below, the second row for pass/fail is blank due to the change failing in testing.
Date of Completion – Date Change was Complete
This is the date of completion of the change. Not the date of the change being pushed to systems. Not the date that all changes are implemented. This is the day that the changes are implemented, operational testing complete, results of change are provided to concerned parties, and the environment is in a stable status. If you are required to collect evidence for regulatory or business compliance, that needs to be completed before the change is considered to be complete. In the example below, the second row for Date of Completion is blank due to the change failing in testing.
Notes – Any Pertinent Information
This cell will contain any pertinent information about the change beyond what is already required in other cells. In the example below, there’s a short note as to why a change was not completed, which was due to the test workstation crashing. This cell could include other information such as who did the patching if they weren’t the Change Manager, if user accounts/password were modified, or a vendor ticket opened for support.
Change Management Workflow Example
Below is an example of the change management workflow. Click the image to view a larger version.
Stay Up to Date with SMB Cybersecurity Resources: Join Dragos OT-CERT!
Dragos OT-CERT offers FREE resources to help SMBs build their own manufacturing / ICS / OT cybersecurity program without hiring any cybersecurity experts. OT-CERT membership is free and globally available to OT asset owners and operators. Resources are oriented toward small and medium businesses and resource-challenged organizations with OT environments that lack in-house security expertise. Members have access to a growing library of resources such as reports, webinars, training, best practices blogs, assessments, toolkits, tabletop exercises, and more.
Currently available resources include:
- OT Cybersecurity Fundamentals Self-Assessment Survey
- OT Asset Management Toolkit
- Self-Service OT Ransomware Tabletop Exercise Toolkit
- Collection Management Framework for Incident Response
- OT Cybersecurity Incident Response Toolkit
- OT Data Backups Guidance
- Host-Based Logging and Centralized Logging Toolkits
- Access to an introductory ICS/OT cybersecurity module in Dragos Academy
If you haven’t joined Dragos OT-CERT don’t delay! Membership is open to organizations that own or operate a manufacturing / ICS / OT environment. Please join and spread the word to your community and supply chain so we can all work together to raise the security posture of the entire ecosystem – we are only as strong as our weakest link.
We look forward to working with you to safeguard civilization!
Ready to put your insights into action?
Take the next steps and contact our team today.