If you were to ask me what my favorite movie of all time is, the obvious answer would be Ghostbusters. If you were to ask me what my favorite dinosaur related movie is, it’d be 1985’s classic movie Baby: Secret of the Lost Legend starring William Katt. But if you were to then ask me what my favorite movie portraying the consequences of an isolated location losing control over critical Industrial Control Systems (ICS) the clear answer is 1993’s Jurassic Park starring Tyrannosaurus Rex as the erstwhile protagonist, “Tyrannosaurus Rex.”
In this post, we’ll review some of the reasons why Jurassic Park is one of the best movies we can watch to learn about ICS security, learn a little about the types of systems John Hammond’s island of broken dreams relied on, and ask ourselves why John Hammond had such a problem with his grand kids.
Background
You’ll recall from the plot of Jurassic Park that the island was created by a navel gazing elderly man and his production company, InGen. All of the systems on Jurassic Park needed to run independently of any other systems existing off the island. Originally built on a fictional island 300 miles off the cost of Costa Rica, Isla Nublar needed to be completely self sufficient in terms of its power generation, distribution, water, and all other life-sustaining operational systems. While these systems were (mostly) functional, the inherent flaws in the systems that had been built, as well as a cascade of other failures ranging from technological to human resources created a cataclysmic situation where loss of life was imminent. Including the lives of Hammond’s grandchildren which he questionably allowed to be sent to his half-finished island. Know Thyself
Everyone has a breaker panel in their house, but chances are that yours doesn’t look much like the breaker panel on Isla Nublar. After the surviving inhabitants of the Visitor’s Center control room make the fateful decision to restart the system by throwing the breaker, they get more than they bargained for when they try to restore power to the island. Dr. Ellie Satler volunteers to go across the compound to manually restart the main island breaker when Mr. Arnold fails to return.
A Westinghouse model SPB65, very similar to the breaker used in the substation bunker in Jurassic Park.
In the movie, the unforeseen consequences of opening the main circuit breaker cost the survivors of the dinosaur escape dearly. The second order effect of throwing the breaker was that all of the other systems that relied on computerized control had to be reset after the power came back on.
There are two sides to every incident; when the system has gone down, and when it is in recovery. Both states may require different approaches to make sure that the integrity of all crucial secondary systems is intact. It’s easy in the wake of an incident to forget that there are additional risks to be addressed when the system is on the road to stability.
You aren’t out of the woods just because you’ve addressed the cause of the system failure. Who is helping recover your systems? Are they trustworthy? Is it possible that a required patch to address the original crash could introduce more instability? Will you be unjustly depending on the IRIX knowledge of your terrified granddaughter to fix the mess you’ve made? Will you, John? Will you!? Don’t Underestimate Murphy
Beneath the edifice of a cool dinosaur viewing experience is a massive amount of systems that the audience never gets to see on the island. While the miracle on display on the island is obviously the dinosaur exhibits, another technological marvel is just beneath the surface in the form of cutting edge industrial control.
Industrial systems are a finely coordinated ballet of physical transfer. Whether that transfer is electricity, candy bars, or refined oil, these processes are very sensitive to changes in time, temperature, speed and any number of other factors. How violent these systems can be if and when they fail is also dependent on a variety of factors.
While one important observation could be that insider threat can cost lives in Operational Technology (OT) environments as evidenced by Jurassic Park’s antagonist Dennis Nedry, another observation is that Murphy and his infamous law was always lurking in the background. In the case of Jurassic Park, Murphy’s Law manifested itself in the form of a tropical storm that swept across the island which caused the park to be minimally staffed and ultimately hurt recovery efforts.
While Murphy’s Law is a hard factor to account for, you don’t have to be taken completely by surprise if the fates conspire against you. Making sure certain backup plans are in place like hot and cold sites to shift critical functions to in a disaster can be helpful in avoiding a catastrophe. In the worst case scenario, those plans could be the only thing standing between, say, your grandson’s hands and a perimeter fence carrying 10,000 volts of electricity…John…
Dragos proactive services staff deals with the ‘what if’ scenario of Murphy’s Law to help system owners prepare for the eventuality of a system owner’s ‘worst day ever’ while Incident Response staff deal with the aftermath of that worst day ever actually occurring.
Don’t Put Off “Crown Jewel” Analysis
If you were to ask most people what the central problem presented in Jurassic Park is, they would quickly and confidently tell you that it was the dinosaurs being loose. That’s a fair answer, but a root cause analysis would show that the reason the dinosaurs got out was primarily because of a loss of power. Indeed, most of the movie’s plot revolves around solving the problem of not having power, but it’s not just this issue that would have been a threat to Isla Nublar. Even inconsistent or intermittent power delivery could have been the cause of a dinosaur escape.
What systems are most important to the functionality of your continued operations? If you had to answer this question during a crisis, would you be able to answer it confidently and completely under a time crunch? Threat Operations Center professionals at Dragos Inc. routinely perform crown jewel analysis for our customers to help system owners understand where their potential exposure is.
Hammond and his team clearly didn’t ask themselves this question at any time prior to the events in the movie. Perhaps they didn’t have time, maybe they weren’t comfortable doing this level of critical analysis of the systems they had already built. Whatever the reason may have been, the result was that the continued safety of their guests was put at risk by failing to identify ahead of time which systems were essential to dinosaur containment.
A good crown jewel analysis goes a long way in identifying which systems absolutely positively must be protected at all cost. This may be be due to the fact that the absence of such a system could lead to immediate loss of production, or it could be to mitigate loss of human life. If John Hammond had truly ‘spared no expense’, he would have paid a consultant to tell him where his company’s biggest pain points were instead of making his Tim and Lex unwilling beta testers.
He was a big dreamer. Tragically though, not a big enough fan of critical systems analysis. There’s No Substitute For Redundancy
If John Hammond had spent a little less time gushing over his executive chef’s Chilean sea bass and a little more time talking to his system architects and doing proper crown jewel analysis, he may have been able to build in the kind of redundancy his island needed. You know, to keep people from getting killed.
What’s clear from the events of the movie is that Hammond’s team built in an awful lot of systemic bottlenecks. Things like letting vehicles stall out on a tour route because no thought was put toward a dedicated backup power distribution system for the tour vehicles, stranding them in front of the most deadly creature for hundreds of miles (thanks for that by the way, John) shouldn’t be happening on dinosaur island.
I’m not saying that John Hammond didn’t care about the safety of his grandchildren. I’m not saying that. I’m only asking, “did he care enough about the safety of his grandchildren to put a little more than chance between them and the teeth of the vicious killing machines he wantonly and irresponsibly created?” The answer to that question is clearly and resoundingly ‘no’. In fact, not even enough to plan in some resilient systems.
Redundancies can be a ‘nice to have’ if there’s extra money in the budget, but redundancies for critical systems should always be considered an important part of the most liable production systems. What would have happened if an uninterruptible power supply could have powered the velociraptor and T-Rex enclosures for even one hour? Would Nedry’s insider attack on the island’s primary systems have led to a dinosaur escape? Probably not. Automation Isn’t Easy Or Cheap. Neither Is Good Help.
John Hammond wasn’t messing around when he got to Isla Nublar. He wasn’t running some two-bit flea circus like last time. He had his own company and he was ready to fully fund this idea that was so absurdly bad that 3 of the 4 experts he hand picked to come to his island to tell him what a good job he did ended up castigating him for being a starry-eyed octogenarian who needed to pull it together before he got people killed (he should have taken their advice).
We constantly hear throughout the story about how Mr. Hammond “spared no expense” on anything on the island. We can see how much the island relies on computerization. Everything from DNA sequencing to automated dinosaur feeding was performed by 7 CM-5 supercomputing units located in the Visitor’s Center. Which is why it’s a bit surprising that when it came to possibly the most critical job role he was hiring for – the guy who makes the whole island work – he went with the lowest bidder.
@hexadecim8 with the homies
Mr. Hammond put all of his trust in a man named Dennis Nedry to run his automation operations. Nedry was essentially in charge of making sure that every PLC on the island was orchestrated back to a unified system in a single room on the island. This is a highly crucial position that requires someone with a great deal of experience, primarily because of the amount of trust that must be given in order for whoever gets hired into this job to be successful.
It is in this one man’s hands, not an entire team of automation experts, that Mr. Hammond leaves the entirety of his hopes and dreams for his island. Oh, and also the lives of his grandchildren (seriously, what was he thinking bringing kids to this death trap?) In one lucid moment, John Hammond even comments that Dennis has “butter fingers” which as far as I can tell is the most scathing criticism he’s ever leveled against anyone in his whole life.
The reality is that if you shoulder a great deal of responsibility onto a team consisting of one unhappy employee, there’s a good chance something untoward is going to happen to you and your company. Vetting human resources is a non-trivial part of any business, but it’s even more important to get it right when so much is riding on the success or failure of that one person’s expertise alone. We can learn from John Hammond’s mistake and hire a team of qualified individuals. We can yet improve on that decision, and institute a strict separation of duties policy to ensure that all of the most important operational information isn’t easily available to just one person with questionable motives.
Conclusion
Being an OT system owner takes a lot of planning and forethought. It also takes a lot of care and respect for the systems that make everything happen. They say an ounce of prevention is worth a pound of cure, but at Dragos we believe in having both available whenever they’re needed. Using Dragos proactive services, and Incident response will help prepare your system for everything you can account for, and help get back to normal if the worst day ever happens.
Ready to put your insights into action?
Take the next steps and contact our team today.