Refuted by Default: Dragos’ Methodology for AI-Driven Vulnerability Detection in OT Security Software

Dragos joined Project Glasswing to apply Claude Mythos Preview against its own systems, hunting for novel vulnerabilities the way an AI-equipped adversary might. The premise is uncomfortable on purpose. Dragos software sits inside industrial control networks, watching the systems that move power, water, and gas, so a defect in the tooling is a way into the defenses around critical infrastructure. Building an adversary competent enough to find those defects means building the exact capability you would least like someone hostile to have. Dragos would rather own it than wait to meet it. If a frontier model can find exploitable flaws in OT security software, Dragos wants to be the first one pointing it at its own.

This post is not about the program. It is about the methodology Dragos built to make the results trustworthy, because that turned out to be the actual engineering problem.

Point a frontier model at a codebase and ask it to find vulnerabilities and it will find vulnerabilities. Hundreds of them. Most of them are wrong: sinks defended one layer up, dead code paths, auth gates the model never read, “attacks” that quietly assume the attacker is already an admin. A tool that ships that noise to engineering gets uninstalled in a week, and it deserves to.

Dragos optimized the whole system for one thing: signal quality. Every finding that reaches an engineer must survive adversarial verification, at three separate kill points, by agents whose job is to kill it. Across Dragos runs, 60 to 90 percent of raw candidates die before a human ever sees them. That mortality rate is the product.

A mortality rate is only worth quoting if the killing is grounded, though, and at first it wasn’t. A verifier handed a bare claim has to rebuild the whole system from scratch to judge it: where the trust boundaries sit, which entry points an unauthenticated caller can actually reach, what the framework already blocks. Ask hundreds of verifiers to redo that derivation independently and the answers drift. One kills a claim because it found the auth gate; the next misses the gate and passes a twin through. The kill rate stops measuring the code and starts measuring which verifier got lucky.

The fix was to stop reasoning file-by-file and start reasoning the way the attacker does. An adversary does not open a repo at a random line. They map the ground first: what is exposed to the internet, what those exposed components trust, which of them touches something worth stealing. Only then do they hunt a specific bug, and the bug they want is the one that moves them one step along that map. Recon, initial access, escalation, lateral movement. The kill chain is an old framework and it is still the shape of the work.

Both halves of the pipeline need that map for the same reason the attacker does. Hunters need it to tell a sink a stranger can reach from one buried behind three auth checks. Verifiers need it to kill a claim by pointing at an existing defense rather than guessing at one. So Dragos builds the map before anything hunts, and writes it down.

That map is the threat model, and it is deliberately shaped like a kill chain viewed from the defender’s chair: trust boundaries, entry points, privileged operations, cross-boundary data flows, and, critically, a catalogue of mitigations already present. The first four sections answer, in order, the questions an attacker works through. Where does this system start trusting me? What can I reach before it does? What is worth reaching? Where does my input travel once it is inside? The mitigations catalogue answers the defender’s question instead. It exists so verifiers can kill a finding that is defended a layer up without re-deriving the defense from scratch every run.

The threat model also partitions the codebase into focus areas so parallel hunters do not overlap, and it persists as a library file that every later mode consumes. It is the artifact that carries context across everything else Dragos built. When a sweep of one service needs to know whether a downstream service trusts the data it receives, the answer comes from that service’s threat model, not from a model’s guess.

With a map in hand, the remaining question is who does the hunting and at what scale. Dragos ended up with four operating modes. They are not maturity tiers; they are modes of operation, and an operator moves between them mid-session.

Whatever the mode, findings move through the same pipeline: model, hunt, refute, prove, judge. Three of those stages can kill a finding, and nothing reaches an engineer unless it survives all three.

Hunt is tuned for recall, on purpose. The red-team agents surface every plausible source-to-sink pair in their assigned focus area and are explicitly forbidden from self-filtering. False positives are expected and cheap at this stage. Precision is someone else’s job.

Between hunt and refute, raw candidates collapse to unique patterns. Hunters tuned for recall rediscover the same weakness many times over; one early run spent 720 verifications on what turned out to be roughly 40 distinct patterns. Deduplicating first means the expensive skeptic stages see patterns, not instances.

Refute does the heaviest lifting, and it has one structural trick: the validator is shown only the bare claim, never the hunter’s reasoning. A verifier that reads a persuasive taint trace tends to be persuaded by it. The Dragos validator cannot be, because it never sees it. It gets a file, a line, and a severity guess, then goes hunting for the upstream guard, the auth gate, the dead path, or the framework mitigation that kills the claim. When uncertain, it defaults to refuted. This one design decision did more for signal quality than everything else combined.

Prove means an observable effect. Survivors go to an exploit-dev agent running against isolated lab environments: docker-compose stacks for individual services, complete platform builds pinned to a shipping release, and isolated replicas of the same appliances, virtual machines, and cloud instances customers actually run. The bar is a proof of concept that reproduces three times from clean state, with no test scaffolding doing the work. Prose arguments are not accepted. Observable effect or it didn’t happen.

Judge is a cynic, and its commentary ships verbatim. The final gate is an agent shown the team’s full argument, whose job is to attack the argument rather than the code: severity theatre, preconditions that quietly assume a privileged attacker, PoCs that prove a different bug than the one claimed, lab-only artifacts. Whatever it writes goes into the report unedited, next to the finding it is criticizing. Engineers reading the reports see the strongest case against each finding alongside the case for it.

Two more stages run after a finding ships, and they are where the leverage lives. A chain-builder returns to the kill chain the threat model sketched and tries to walk it for real. It treats validated findings as primitives (read, write, auth bypass, leak, forge, pivot) and searches for compositions that cross a trust boundary none of them cross alone. This matters because it is exactly what an AI-equipped adversary gets for free: the patience to hold forty medium-severity primitives in mind and find the sequence that turns them into a critical. And a variant-hunter takes each confirmed bug’s structural signature and sweeps every threat-modeled repo for twins, which re-enter the pipeline at the refute stage like any other candidate. One proven pattern becomes an org-wide query.

None of the above is worth anything uncalibrated. Before the static modes were used on new ground, each had to independently rediscover known results: Dragos keeps a ground-truth document of previously analyzed, high-severity finding chains in two of its services, and each mode had to surface at least four of six, blind, before it was allowed to widen. The live-target track gated on a cheaper question instead: does the test fixture actually reach the code under test?

That cheaper question produced the most expensive lesson of the program. One run spent 68.7 million tokens across 748 agents and confirmed nothing, because the test identity could not reach the code being tested: every proof attempt died at an authorization gate before the target logic ever ran. With the fixture corrected, a re-run confirmed 12 findings, four of them critical, for 3.7 million tokens. Same candidates, same pipeline; only the setup changed. Setup correctness dominates compute, and a cheap “does this fixture exercise the code path” probe before the expensive loop is the highest-leverage spend in the whole system.

Calibration produced the single most convincing moment of the project. One of the six ground-truth entries was an XXE finding in an XML ingestion path. During a calibration run, a validator did what validators do: it tried to kill the claim, empirically, by actually executing entity expansion payloads against the pinned parser version. The parser disables entity expansion by default and has since several versions back. The finding was wrong. Dragos’s own ground-truth document was wrong. The pipeline refuted its own calibration target, and it was right to.

That is the behavior you want from a verification layer, and it is worth saying plainly: a shipped false positive is a precision failure even when it comes from your own answer key. Dragos corrected the document and now treats “kills the bogus entry” as part of the calibration bar.

The largest run so far took the entire first-party surface of one shipping platform release: every code service and configuration chart the release references, each pinned to the exact commit that shipped, swept as a single deterministic workflow in eleven shards.

Two details in that table matter more than the totals. First, verification is scripted, not vibes: candidates face a skeptic panel and survive on votes, so the refute rate is a measurable, trendable number per run. The panel is not uniform either; rigor scales with severity. A critical claim earns more independent refuters than a medium, and crash-class findings skip the panel entirely, because a reproducing crash is its own verification. The parser-fuzzing track, where most findings were crash-class, had the best yield of the whole program for exactly that reason. Second, the sweep consumed the previous wave’s refutations from a shared state store, so wave N does not rediscover what wave N-1 already killed. Cross-run memory is what makes repeated sweeps converge instead of churn.

The ten cross-service chains then went to prove: one exploit-dev agent per chain against a dedicated sandbox running the same release build the sweep had read, each agent starting from nothing but the entry privilege its chain claimed, followed by one cynic per chain judging the finished argument.

The two most useful numbers there are the smallest ones. The sweep had rated eight of those ten chains critical. After proof and judgment, two were critical, seven were high, and one was a medium: its second hop depended on a proxy behavior the deployed configuration does not actually permit, and it failed against the running system in a way no amount of source reading had caught. It shipped describing the effect that was genuinely demonstrated, which was real but much smaller. And exactly one of the nine survivors shipped as written; the other eight came back with specific, evidence-cited corrections to their own narratives. “Revise” is the normal outcome, not “ship.”

That is the argument for keeping prove and judge as kill points rather than formalities. Static analysis, however adversarially verified, still reasons about a model of the system. The running build is the system, and it gets a vote.

Those two tables are one sweep feeding one proof cycle, which is two of the four modes working in sequence. The other two produce their own volume. The live-target track alone worked 1,174 raw findings down to 313 deduplicated tickets, 86 of them proven with a working PoC on the appliance, across roughly 78 million tokens, with six kill chains attempted and three confirmed, two of those critical. All identified vulnerabilities were remediated in the normal Dragos patch cycle.

Priced side by side, the tracks are not interchangeable. The deep-dive squad runs cheaper per verified finding than the sweep, and the sweep covers ground the squad never could. That trade is the point of having modes instead of a tool.

The engagement pipeline answers “what is wrong with this product.” The question developers actually have is smaller: “did the work I just complete introduce a vulnerability?” So Dragos packaged a structurally reduced profile of the same pipeline as a Claude Code plugin, scoped to the developer’s diff plus its blast radius, with the same refute-first discipline and a single cynic pass on survivors.

Dragos calibrated it the same way, just diff-shaped: the team rebuilt known vulnerabilities as plausible-looking feature branches and ran the review blind, alongside distractor runs on clean merged PRs to measure the noise floor. The gate required parity with the full pipeline’s kill rate and a noise floor of at most one defensible finding per clean run. It passed at a median of about eleven minutes and five dollars per run, which is the envelope that makes “run it before every PR” a habit rather than a policy.

The oldest complaint in security is the asymmetry: the attacker needs one bug, the defender needs all of them. Every prior generation of tooling nibbled at that imbalance without changing it. Static analysis found the patterns it was written to find. Fuzzing found what it could reach. Pentests found what one team could cover in two weeks a year. The gap between “what we checked” and “what an adversary with unlimited patience would try” stayed wide, and everyone in the industry quietly priced it in.

This is the first methodology Dragos has used that feels like it actually bends that curve, and the reason is home-field advantage. An adversary pointing a model at Dragos products works from binaries, public docs, and inference. Dragos works from source, design documents, a threat-model library, and live environments it can stand up and tear down at will. The same model that amplifies the adversary’s probing amplifies that context far more. When both sides get the same engine, the side with the better fuel wins, and nobody has better fuel for finding bugs in Dragos code than Dragos.

Running a harness like this against your own code first is, bluntly, a missing hardening step. The industry already accepts fuzzing, sanitizers, dependency scanning, and code review as table stakes, and each closes a class of gaps. None of them closes the class an AI adversary is best at: logic flaws in trust boundaries, handoffs where one team’s output becomes another team’s unvalidated input, and compositions of individually tolerable weaknesses into chains nobody designed for. Those are exactly the gaps this pipeline hunts, with an attention span no human review cycle can sustain. Every chain Dragos finds and breaks in a lab is a chain that no longer exists for anyone else to find, and unlike a pentest report, the capability does not expire when the engagement ends. It reruns on every sweep, every diff, every new repo added to the threat-model library.

AI-equipped adversaries are coming for critical infrastructure whether or not defenders modernize. The only honest way to prepare is to be that adversary against yourself first, with a verification layer harsh enough that you believe what survives. Do it before you ship, do it on every diff, and the gaps an attacker’s model would have found are gaps that were closed in a lab months before anyone hostile went looking. That is what hardening is supposed to mean.

The adversary Dragos is modeling does not get tired, does not skip the boring focus areas, and composes small primitives with perfect recall. For the first time, neither does Dragos.

Contact Dragos Today