In our previous blog, “Value of PLC Key Switch Monitoring to Keep Critical Systems More Secure,” we discussed the importance of monitoring Programmable Logic Controller (PLC) key switch positions and subsequent PLC mode changes from within the PLC itself. We provided an example of a programmed AddOn (AOI) Rockwell-specific implementation that leverages the module attribute modes to extract the PLC operation mode through the Get System Value (GSV) function and the use of the built-in tags offered in the RSLinx Rockwell tool. We then provided recommendations for monitoring and alerting PLC operation modes changes in a Human Machine Interface (HMI). Rockwell Automation is a Dragos partner and we focused on Rockwell CIP as an example because of its pervasiveness in multiple industrial environments.
Despite the importance of developing code detection within a PLC and programming displays and alarms on an HMI, these security controls are not entirely foolproof in ensuring the safety and reliability of critical operational technology (OT) PLCs. This is especially true if an adversary abuses open connections or trusted paths. These could be through downloaded firmware to a device that acts as a rootkit and modifies the key switch memory values and locations, such as what occurred in the TRISIS attack. Another example includes an adversary-in-the-middle attack on an OT protocol, such as Common Industrial Protocol (CIP) or Modbus – both are well documented and could be performed by a studied individual.
In this blog, we discuss the value of using the Dragos Platform to effectively detect and respond to configuration changes in network-connected PLCs. We outline the information captured in the Dragos Platform and how a security analyst can use this content to investigate and remediate PLC key switch status changes.
Weaknesses In Protecting PLCs with OT Firewalls
To mitigate some of these attacks, OT-protocol application-specific firewalls can be inserted between the PLC and device (from a protocol perspective) that reads or writes values to the PLC. The interfacing device could be an HMI, Distributed Control System (DCS) controller or server, Supervisory Control and Data Acquisition (SCADA) server, Remote Terminal Unit (RTU), Intelligent Electronic Device (IED), or a myriad of other systems. Once the communication requirements are known, and specific ports, protocols, and application-level commands are referenced and restricted in accordance with the required control scheme, the overall implementation can be made more secure (as shown in Figure 1).
However, the drawback to the example shown in Figure 1 is that there is still a trusted path from the HMI device to the PLC. If this path is not monitored, an adversary can still leverage this pathway for abuse if they manage to compromise the HMI. This is especially troublesome if the PLC key switch is left in the Remote mode, which is a common finding for Dragos Professional Services teams. An adversary could insert or delete any logic in Remote mode, like deleting a Proportional Integral Derivative (PID) Function Block (FB) or altering Sequential Function Chart (SFC) transitions.
These two functions are essential in industrial environments where continuous control or batch control is needed. The PID controls continuous processes or variables, such as temperature control in a furnace or flow control in water pump stations. On the other hand, SFC is used mainly for batch control, where the sequence controls the operations of interconnected equipment or systems such as brewery recipes or unloading a coil in a hot rolling mill. We provide more detail about these two functions below:
- PID controller adjusts an output of an actuator (e.g., control valves or variable frequency drives) based on the error calculated between the actual feedback value (e.g., flow, level, speed measurements) and the reference value set by an operator or internal logic, such as data coming from sequence commands.
- SFCs consist of a flow diagram comprised of steps and transitions where the step commands actions to systems. For example, an SFC step could start pump X or set a PID reference value. Transitions are the conditions that allow the next steps or actions to continue, such as pump Y is running or a PID setpoint is accepted and in operation.
As described, these two functions are the core of control strategies in many industrial processes. PIDs and SFCs can also interact together. In this way, an SFC step could set the setpoint (SP) on a PID, or the sequence status could be used as a PID interlock. If PID or SFC functions are altered or deleted, the process could develop significant disturbances, or even worse, cause safety incidents due to modified controls operations. In a worst-case scenario, the adversary could also disable complete routines or delete routine calls, affecting the logic interconnections with other routines or systems, interlocks, and data validations, which can cause operational disturbances.
Although the firewall may be blocking known protocol commands, the firewall may not protect against all data reads and writes. This becomes apparent when considering the CIP protocol that bundles data read and write requests into a data blob where the data fields can change between project file or device differences. This makes creating granular CIP-specific firewall rules that filter read and write data requests difficult, at best. This is not to say an OT firewall provides little value to the overall security posture of the system(s), but it does bring to light some of the difficulties when selecting a firewall for specific OT protocols.
Latency induced by traffic inspection can also cause havoc on specific high-speed processes where coordination and communication between machines can be in the single-digit millisecond range. Thus, a firewall placed between PLCs in a time and motion critical process may render the system(s) noncontrollable.
Benefit of Detective vs. Preventative Security Controls
Many OT security professionals are familiar with the phrase: ” Prevention is ideal, but detection is a must.” Regarding protecting Crown Jewels, such as critical PLCs, this means that if asset owners and operators cannot fully protect these devices with firewalls, they must monitor them. This is where Network Security Monitoring (NSM) comes into discussion.
When leveraged at lower levels in the Purdue Model, OT-specific NSM solutions can monitor critical east-west system OT protocol-specific communications to detect configuration changes, such as a PLC key switch, and alert security engineers and Security Operations Center (SOC) teams. Implementing passive NSM solutions as a means of detection is essential to provide insights into trusted pathway abuse scenarios, among many others.
PLC and network-based detections provide an analytical depth of coverage when applied to recommendations outlined in our last blog, where we discussed the importance of leveraging PLC functions to detect the key-switch position changes. Figure 2 shows a Dragos sensor, which is part of the Dragos Platform that includes a sensor and sitestore combination used for monitoring OT environments.
However, conventional rack-mounted sensors simply do not work in areas where PLCs reside since the OT systems are often near the plant or factory floor or in remote locations such as within pipeline pump stations. With Dragos DIN-rail mounted sensors, once difficult to near impossible OT locations and networks can be monitored using the same technology stack as the larger sensors but tailored for smaller footprints found within automation cabinets.
How the Dragos Platform Detects Key Switch Configuration Changes
Detecting mode changes correspond to an environmental configuration change with the Dragos Platform. The configuration change is one of the four types of detections the Dragos Platform uses to organize notifications, developed from Dragos’s “Four Types of Threat Detection” whitepaper. The other three are behavior, indicators, and modeling.
Configuration change by itself is not malicious; it is just an indicator that something changed in the monitored network. Provided with enough context, however, an asset owner or operator could quickly determine if this change was part of a managed and planned maintenance or project or not. If the latter, the event turns into an incident and follows the asset owner or operator Incident Response Plan (IRP).
For Rockwell PLCs in particular, the Key State detection capability comes from monitoring the CIP commands flowing across the network, shown in Figure 4.
Those who have had the opportunity to take part in a Dragos Capture the Flag (CTF) or SANS CTF event – including ICS/Grid Netwars — may have seen PLC attributes such as manufacture, model, and firmware while digging into the provided PCAP artifacts for those respective questions. Players are often surprised when they uncover these rich data sets that show how much OT device information travels across an OT network. Nevertheless, the CIP protocol is complex, and many of the interactions with CIP will not be observed or understood in Wireshark alone as the dissectors are simply not available for manufacturer-specific implementations.
CIP Protocol Inspected by Dragos Platform
This is where an OT-specific product, such as the Dragos Platform, brings significant value. The Dragos Platform examines CIP fields, particularly the CIP Extended Device Status in the Identities Path, to determine PLC Key State. As a reference, the following Wireshark display Filters show all CIP identities traffic with a Run or Program PLC states, respectively.
cip.id.ext==0x00000006 or cip.id.ext==0x00000007
Furthermore, the Dragos Platform records the values indicated above and many others to create asset inventories, identify vulnerabilities, and create configuration notifications such as the Key Switch change detection for all discovered PLCs in the environment. Regarding vulnerabilities, the Dragos Platform compares detected firmware versions to the Dragos vulnerability database.
The PLC key switch state change can come from two different ways. First, an engineer, or adversary, can remotely cause a change by requesting a Start (Remote Run) or Stop (Remote Program) using the CIP protocol. Note that these commands work only if the physical key switch is in the Remote mode on the PLC. In Wireshark, these commands can be identified using filters CIP.SC==0x6 for Run and CIP.SC==0x7 for Stop, as shown in Figure 5.
The other detection is when an HMI program is using CIP Identities and requesting device status. Using this, the PLC informs the HMI or Workstation when the Physical key position has changed. This is a great indication to notify if a technician is working on the PLC, for example, and to track PLC key status throughout the OT environment. Hopefully, keys are left in the local Run position unless there is a valid and documented reason for leaving them in Remote.
Using Dragos Platform Notification Manager
As discussed previously, the Dragos Platform leverages extended device class values to determine the Key State status accurately. These status changes are also identified on the Dragos Platform as a Configuration Change severity 1 notification. Notifications with severity 1 indicate that it is something to be aware of for a baseline setting. However, depending on the asset owner and operator context, the notification value can be raised, such as if PLC mode changes occur during running operations.
Figure 6 shows how the Platform displays the notifications for PLC Status change as severity 1, the source and destination IP addresses, and the source of the change (i.e., from the PLC [physical key] or engineering workstation [selection of remote modes]).
When a PLC Status Change notification fires, the OT security analyst can dig further into the details of what systems were involved, which zone the systems are located, and when the notification was first detected and last detected. This all provides rapid situational awareness.
Additionally, the notification describes MITRE ATT&CK Tactics, Techniques, and Procedures (TTPs) and ICS Kill Chain phases and a view into the dataset that the notification fired, as shown in Figure 7.
Leveraging Dragos Platform Playbooks
Armed with this information, an informed analyst who intimately knows the OT environment could be ready to act. On the other hand, they may require additional assistance to determine what actions to take next. This is where notification-specific playbooks can be leveraged to guide OT security analysts to dig further into the datasets beyond the detection information alone.
As an example, the Dragos Rockwell PLC Keystate Change Playbook, shown in Figure 8, provides background information on the detection reasons, how the PLC mode could have changed, and why the status could have changed, which could be appropriate during maintenance windows or concerning by adversarial manipulation.
Dragos Playbooks included in the Dragos Platform also include ordered tasks that guide users to run specific Query Focused Datasets (QFDs) which enable further analysis and support incident response.
Preventing Threats is Ideal, but Detection is a Must
Understanding and detecting changes to critical OT devices is necessary to ensure safe and continuous expected results of OT environments and their underlying processes and functions. Regarding PLCs, this includes ensuring the Key States are monitored. This may include leveraging PLC detection code combined with HMI alerts to notify operations, understanding that it is only part of the overall detection strategy. Products such as the Dragos Platform provide greater system(s) context and increased detection capabilities beyond what open source tools or internal PLC routines can offer.
Remember, prevention is ideal, but threat detection is a must. With this in mind, we as an OT security community must focus on detecting adversaries’ activities and capabilities within operational technologies. Only with this visibility and detection capability will we be assured that our ICS/OT systems are safeguarded for their intended operation.
Ready to put your insights into action?
Take the next steps and contact our team today.