My So-Called Industrially Complex Life
As the war no one notices or cares about drags on, countries are beginning to establish task forces, departments, and entire ministries to foster awareness among their economic sectors—such as chemicals, civil nuclear, 5G networks, oil & gas, and healthcare—most at risk for attacks that would disrupt everyday life and destabilize entire economies. This is not hyperbole from the arc of a Marvel film; these are oft-stated objectives of our adversaries, the usual suspects.
For many commercial or agency threat intelligence groups, the goal is not only to never be noticed by the enemy, but also to never be noticed by users. Protecting users’ ability to engage in social media, global gossip, and all things consumer might seem frivolous and geopolitically absurd, but that is in many ways what we are trying to preserve—modern civilization, faults and all. It also helps to preserve the multi-trillion-dollar economies that underwrite the frivolities. As I can attest from living in Pacific Gas & Electric’s public safety power shutoff zone, keeping the lights on is one of those frivolities.
What is critical infrastructure, how do enemies attack it, and how do we keep attacks from happening? Countries such as the United Kingdom have a sensibly titled Centre for the Protection of National Infrastructure with a website that documents exactly what Her Majesty considers critical. In the United States, the FBI, Intelligence Community, and Departments of Homeland Security, Defense, Interior, Energy, and Commerce all have their own constituencies.
History reveals some awkward wrinkles in the delicate balance between industry and government, and even within governments there’s an instinct towards tribal behavior. Anyone remotely familiar with the military and industrial complex that has been entangled since WWII—possibly since the US Civil War—was not entirely surprised that the September 11 attacks had something to do with the CIA, NSA, and FBI not always being on speaking terms. Many things did change after 9/11, but historic reflexes relating to cooperation, whether in agencies or in commercial organizations, can be a real impediment towards modernization. There has been good progress, but whether or not it is one big happy family is a little harder to tell. Unfortunately, on a day-to-day or a contract-vehicle basis, it’s not always; on a year-to-year basis, yes, it is getting better.
It’s also no secret that the US economy isn’t really the purview of the US federal government. The banking industry is run by Wall Street. The Federal Reserve Bank system is not a federal agency, but does have a complex relationship with Wall Street and the Treasury. The unregulated math-majors frenzy that led up to the 2008 “glitch” forced the US federal government to step in and drop their multi-trillion-dollar credit card on the table; this was not their first instinct, nor was it the result of the first TARP vote.
In the US, it’s also not natural for the NSA to oversee the protection of cell tower infrastructure belonging to a company majority-owned by a German telco that is partially owned by the German government. Nor is it the federal government's natural instinct to protect ExxonMobil’s pipelines and refineries while their raw and refined products fly around the Montana countryside. There is a learning process around threat intelligence, and many corporations still aren’t concerned about it—they think that the government should just manage national defense, whatever that might entail, and let companies conduct commerce. This would be great, except that our enemies don’t distinguish between Old Navy and the United States Navy, or between Exxon and Zappos.
Oil is the New Oil
I grew up in the Canadian oil patch. It is a little-known fact that all of the raw refinery products from Alberta destined for eastern Canadian refineries flow out of Canada and into the United States—through Montana, North Dakota, and Michigan—and then under the river back into Canada. A set of power lines hangs across the St. Clair and Niagara Rivers between Ontario, Michigan, and New York. Anyone who witnessed the northeast cross-border blackout in 2003 understands that U.S. and Canadian critical infrastructure is linked for many practical reasons, but now both sets of government agencies, NGOs, and corporations must figure out how to keep that infrastructure from being hacked not only by the usual suspects, but also by an increasingly dynamic Mother Nature.
The first step in protecting infrastructure is to become situationally aware. We must not only detect and protect infrastructure from pesky actors but also ensure that it can continue to operate in an ice storm. The only difference between bad actors and ice is intent. The mantra for operational vigilance is to assess, predict, and protect—just not always in that order.
I have been around datacenters for most of my career, but now focus most of my time outside them which informs my role as a threat, situational, and edge technologist at Confluent. Critical infrastructure rarely lives in a datacenter. I love to put Kafka in a backpack and help teams enable their awareness modernization for realtime anomaly detection or model training. Assessing and protecting pipeline pumps in Yellowknife or cross-border isolation transformers in the middle of the ice storm almost always means getting out of the truck, not just out of the datacenter.
Operational Technology – The Fine Print Behind the Internet
The Internet of Things assumes that many industrial sensor fabrics are connected to the internet or even that Ethernet is used as a communications technology. This is especially true in the deployment of industrial IoT. All forms of industrial IoT or process control automation fall into the larger category of operational technology. In many industries, industrial IoT is still an emerging technology, so assessing and protecting legacy OT infrastructure is often the first step. The price we pay for instantaneous networks spanning the globe is that an enemy, once sequestered in a two-story walk-up somewhere in eastern Europe, is now 50 meters from your firewall or unprotected MQTT relays in northern Norway. “Stay the hell off the internet” might seem like a Luddite’s favorite laptop sticker (found on their Honda generator), but they can’t get to you if they can’t get to you.
Though many bits of the economy must be accessible via the internet, other bits need to be isolated or “air gapped.” Physical isolation is a simple and effective defense, but it makes it harder to assess, protect, and predict failures. An enemy can still find IP-centric ways to infiltrate command and control that might be accidentally exposed to the internet, so isolating networks that can still be accessible is not common in the industrial world, but it is common in the world of classified computing.
This Blog Does Not Exist
In the class/unclass world, the “hard line” is cut and networks are isolated from even private intranets within a given building. In these conditions, the movement of data requires a hardware data diode, usually an opto-isolated set of circuitry that only implements unidirectional UDP ports. This one-way UDP stack transfer requires modifications to transfer protocols. Although this has been used within the DoD for years, organizations now need to protect and maintain their own isolated networks. Confluent developed a data diode connector for class/unclass topology, but we are finding that an “off-label” use is in air-gapped networks that still need never to be connected. Some power-generating stations—the fissionable kind—use data diodes for isolation, but any piece of critical infrastructure with this level of isolation can deploy data diodes to permit telemetry to flow over to analysts and data scientists sleeping on the datacenter floor waiting for the data to arrive.
Some of our customers must protect rail track, power transmission, and water supply lines since all of these are easy candidates for the critical list. Air-gapped sensor fabrics exist in refinery cracking towers and manufacturing floors in addition to those pipeline pumps in the countryside. Deploying small, compact instances of Kafka enables customers to acquire telemetry persistently and immutably in places where it had not been practical in the past, where nothing was being collected. Kafka allows a variety of timestamps, so sensor array data can be collected in millisecond phase; this can be useful if there are cascading sub-second events, or when subtle changes in the temperature across a set of 4,000 sensors indicate an operational anomaly of interest. Kafka at the disconnected edge is not a pristine-datacenter, exactly-once gig. In many cases, it’s more like even once, so having a simple standalone Kafka server running on a Mac mini, 15W Arm server, or Raspberry Pi enables Kafka-in-a-command-post, Kafka-in-a-pickup-truck, or even Kafka-in-a-backpack where forensic telemetry can be brought back to the post and curated for instant awareness or staged and forwarded up to data scientists.
Many pieces of critical infrastructure live far beyond a datacenter and often analysts have to go out with the truck to take action. For predictive and protective aspects, this data can be blended with other sources from the edge and the core. This blend can be used for deeper analysis or used for ML model training. These models can be deployed back out to the edge to improve the operational reliability through the use of streaming ML. Streaming ML, aka machine doing, is the art of deploying trained models onto realtime streams of data.
Game of Drones
Inspecting power transmission lines usually involves a helicopter, a crew, and some random amount of cash to keep the crew flying. Drones have revolutionized filmmaking, but they’ve also made it possible to inspect hundreds of miles of PG&E wires after a wind event. I live in the mountains above Silicon Valley and the new normal for PG&E customers is extended periods of de-energized grids to avoid another “Paradise event.” After the wind and before the electrical grid crackles back to life, helicopters buzz our house at about 200 feet.
Drone telemetry operations are a form of geo-fenced air-traffic control and the drones will swarm in an operating arena, but even if the swarm is just a couple of drones, both their sensors and their position in the area can benefit from smart streaming when one microservice looks for failing insulators and the other one ensures the drones don’t fly into each other.
Uppercase Artificial, lowercase intelligence
I was born with a rare neurological disease and have been monocular for a long time. My cortex rewired itself to compensate for two-thirds reduction in visual processing. No algorithms were changed. The human brain is not a computer. It is a massively complex, analog signal processing engine and there are no algorithms running anywhere. When postdocs talk about using algorithms to implement feed-forward, hidden-layer methods that can find your grandmother in a master shot of Grand Central Station, they are talking about using algorithms to do what your brain does with greased neural pathways and weighted signaling.
In many ways artificial intelligence, machine learning, deep learning, neural learning, and natural intelligence have all been diluted by marketing departments. But the simple reason that threat intelligence groups who are tasked with protecting critical infrastructure turn to these tools is that they need more sophisticated “algorithms.” ML and AI are a form of dynamically adaptive algorithms; they adapt when the conditions or the enemy insist on being dynamically adaptive. We already know that static code and running queries in SIEMs are no longer effective responses. They’re even less effective countermeasures because it just takes too long. Streaming ML—machine doing after the machine has learned—addresses both of these weaknesses. I don’t know if we are heading towards James Cameron’s Skynet, but computers remain artificially intelligent and there’s still a big gap between good math and good judgement.
Humans aren’t computers. Humans don’t run algorithms. Because the central nervous system evolved to do very different things from computers, there is a complementary division of labor between human doing and machine doing. Getting algorithms running on streams of traffic flying into a Kafka cluster at Netflix velocities means you can track left as soon as your enemy tracks left. Some of our customers are deploying trained models on live streams using a feature in KSQL called UDF. Dynamic algorithms are a welcome tool as the world continues to weaponize data.
When Data is the New Oil
When we sufficiently weaponize data and then go to war with it, data will have graduated to one of those globally precious resources that tribes love to fight over. Long after we use big, fast, and adaptive data to help wrangle the climate, we will still have something worthwhile to fight over: all of that data science, intelligence, and petaflops.
Throughout history, effective military campaigns have always required timely and accurate field intelligence. In the age before messaging at the speed of light (i.e. radio), this meant a horse. Before that, people were running. Today’s battlefield commanders and corporate CISOs now have a nearly unlimited supply of data at their fingertips. Low quality intelligence can lead to poor command decisions, so being able to utilize Confluent’s Curation Fabric underwrites the fidelity of intelligence. “Curation” can range from excluding some columns to extremely complex forms of stream processing using Kalman filters to help prevent the drones from flying into each other.
Data is not intelligence. And more data is not more intelligence. Effective threat intelligence starts with good data hygiene and that always means cleaning up the signal. The ability to sift or curate data into intelligence requires computational skills to get the signals to sufficient fidelity so that an analyst or commander can make informed tactical decisions. Confluent’s Curation Fabric can help commanders, CISOs, analytic models, and SIEMs make better decisions.