Exclusive Webinar:

Join us Oct. 6 as Rockwell Automation & Dragos CEOs reshape the way you approach cybersecurity in manufacturing.

Skip to main content
The Dragos Blog

01.23.19 | 5 min read

Dragos Results of S4 Industrial Cybersecurity Detection Challenge "Contest"

Dragos, Inc.

Overview of 2019 Detection Challenge

Last week, the Dragos team participated in the S4x19 ICS Detection Challenge–a challenge designed to assess the capabilities of industrial control systems (ICS) cybersecurity platforms to detect real-world ICS incidents.

This is the second year in a row S4 ran this challenge. Last year, the challenge was based on a small packet capture of 3 GB that included a baseline capture, and participants could use any product with access to any analysts over two days. This year’s challenge significantly evolved from 2018 and presented a much more complicated attack sequence to better represent a real-world, multi-component attack–encompassing 400 GB of packet captures from a real-world mining operation. This data represented multiple networks spanning the facility with a set of adversaries “injected” by the S4 organizers for participants find, and there was no baseline. This year also required participants to use their own products. Coupled with the new dataset, this year’s challenge significantly tested participating ICS security vendors’ asset identification, threat detection, and response capabilities and pushed them to operate at scale across a large dataset in a short amount of time.

Over 500 hours of effort were put into developing the challenge–transforming it from last year’s challenge–but the final outcome of the challenge turned out to be a product demonstration, as the actual contest was cancelled, due to a lack of participants.

Consequently, a revised detection challenge dataset of 130 GB was created and shared with two teams (out of a possible 20 teams). From this revised data set, Dragos analyzed 140 network protocols–including a mix of control protocols such as CIP, OPC, DNP3, ENIP, and Modbus–mapped 4600 unknown assets, and discovered malicious industrial campaigns through intelligence-driven threat behavior analytics.


The ability to distill 130 GB in such a short time was possible because the Dragos Platform enables data to be processed and analyzed for automated asset discovery and mapping of the environment. The Platform also noted changes in the environment over time and identified threat behaviors to focus analyst resources. Because there was not enough time for a threat hunting team to review page after page of anomalies, the execution of the detection challenge was very dependent upon the Dragos Platform automatically identifying some key behaviors as a basis to begin our investigation. The automated detection was enabled through characterizations of specific patterns of behaviors, as well as by mapping a flow of adversary communications, connections, and failed authentication attempts.

Step 1: Environment Discovery and Asset Identification

The first step in the challenge was to understand the ICS environment. Using the Dragos Platform’s deep packet inspection for industrial environments, the team had in-depth asset visibility–including zones, timelines, and query-focused datasets. From this, Dragos identified over 4600 assets communicating over 140 network protocols, including a mix of control protocols such as CIP, OPC, DNP3, ENIP, and Modbus. The Platform’s zoning capabilities allowed the Dragos team to quickly understand the individual networks, as well as their trust relationships to other networks. This summarized information was visualized in our Platform to quickly begin analysis and exploration of the dataset and to quickly sort and filter information about users, vendors, and specific communications.

The following is a screenshot of the mapped assets.

Environment Discovery

Why this is important:
Detailed asset identification and management provides analysts context of the “who, what, and when,” so they can launch effective investigations into suspicious behavior and spend less time on irrelevant data. While some network activity may be normal, the Dragos Platform provides context beyond anomaly detection to help analysts pinpoint suspicious behavior, dig deeper into the data, and begin triage if the behavior is malicious.

Step 2: Detection of Threats

The next step was to detect threats within the data.  Our team began at the Dragos Detections Dashboard, which identified 26 threat behaviors, 11 deviations from baseline, 51 configuration changes, and 5 indicators. These summarized notifications allowed us to begin piecing together the story. For example, one of these communications indicated lateral movement through credential re-use.

Extracting this summary from 130 GB of data would typically require copious amount of manual effort to process and analyze and would not be possible in the timeframe allocated for the challenge–in this case, the participants in the revised challenge were granted 5 days, and the original design of the challenge was to complete this in 8 hours.

Detection of Threats

With comprehensive context of the ICS environment, Dragos analysts automatically received notifications from the Dragos Detections Dashboard related to HAVEX and GREYENERGY behaviors–both malicious industrial campaigns. It was also possible to note a variety of alerts related to different malware families in the environment, as well as uploads and modifications to controllers–although some were accurately deemed to be false positives.

Why this is important:
Threat behavior analytics identify threats through complex patterns of adversary techniques, tactics, and procedures–because they are developed by practitioners who have already hunted and characterized them. The ability to chain together a series of events into a custom analytic allows for customization of behaviors that have a stronger relevance and fidelity of information, rather than relying on one static, atomic value.

Step 3: Triage

From the highly summarized notifications generated by the Dragos Detections Dashboard, analysts were guided to playbooks for different threats. For example, after multiple HAVEX notifications, analysts were guided to the “HAVEX-Like Activity” Playbook for triage. Playbooks provide step-by step guidance for triage, including scoping of the potential incident, as well its associated notifications. After identifying the investigation scope, the next steps included identification of scanning activity and exfiltration of data indicative of additional HAVEX tactics, techniques, and procedures. (Tracked since 2013, Havex steals intellectual property to support attack development on industrial control systems).

Further investigation of communications indicated lateral movement through credential re-use. The Dragos team was then able to quickly follow the pattern to a host showing indications of a previously known variant of GREYENERGY remote access trojan.

Activity Detected

Critically important to the investigation, the Dragos Platform not only analyzes through deep packet inspection and tracks behaviors, indicators, and configuration changes, but also serves as a logging platform where analysts can search for specific communications between hosts–including the ability to export files moving across the network, investigate commands being sent to controllers, and track user accounts authenticating to hosts. This ability to dive deeply into the dataset natively prevented the need for network tools not designed for such large datasets and allowed the analysts to remain focused on analysis, rather than processing data in order to perform analysis.

The case management system enabled analysts to track each other’s notes, attach files, and  keep work consolidated in one spot. This, along with the tried-and-true whiteboards, allowed the Dragos team to collaborate effectively over a complicated attack sequence.

Case Management System

Why this is important:
Context amongst the chaos of cyber events or incidents is the key to reducing analysts’ time spent figuring out what’s going on and how they should respond. Intelligence-driven context matched against system events, notifications, and investigation playbooks can quickly empower analysts with step-by-step guidance throughout the incident response process, such as scoping of the incident and understanding its associated notifications.


The above approach summarizes a few of the major cyber events Dragos discovered throughout the 2019 S4 Detection Challenge. Our team carries this approach throughout its real-world ICS security monitoring, threat detection, and incident response. We understand system owners manage large amounts of assets, and identifying threat behavior associated with those assets can be difficult–which is why we built the Dragos Platform to empower analysts with the context and tools to comprehensively identify assets, effectively detect threats, and efficiently respond to incidents.

To learn more about Dragos’ findings from the 2019 S4 Detection Challenge, watch the video from our S4 main stage presentation:

Ready to put your insights into action?

Take the next steps and contact our team today.