I regularly work with these sorts of water districts (larger, better funded ones as well). In reality, some of these small districts may only have 2 or 3 SCADA operators on staff. Sending them home with a pager, a tablet, a VPN password, and some overtime pay is a lot easier to get past the city council then taking on another two employees to cover the night shift for those rare events that need to be handled ASAP.
I could share some real horror stories, but it wouldnt be professionally appropriate. Suffice to say, this story did not surprise me at all.
I reject this line of thought.
A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades.
They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances.
As for telemetry ...
I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?
I like your thinking and I try to espouse it myself (keeping things as simple as they can be-- keeping "technology" out of voting, not connecting things to networks that have no business being connected, etc). Short of a Battlestar Galactica-type "our machines rise up and try to kill us" event, though, I don't think the average person will ever understand the vulnerability inherent in networked computers or the risk/benefit tradeoff of connected vs. disconnected systems.
Even down at the level of local politics in a rural setting the "optics" of bringing technological solutions to bear on problems is seen as forward-thinking-- particularly when it "saves" the taxpayer money. I can't imagine trying to convince a local water board that moving away from a PLC-based system with a remote support vendor world fly, even citing this example.
This event will be another opportunity for more security vendors to cite in case studies justifying their products. More layers of garbage will build up on a foundation of protocols and design philosophies that grew up in an era of disconnected systems with lower stakes and a less complex threat model.
I don't see big money to be made in providing sensible levels of connectivity and security to this kind of infrastructure. I don't see industry stepping-up because of that. Maybe regulation is the answer, though I'd just expect regulatory capture to take over, and have it become another "PCI". Maybe a lot of people have to die before society takes it seriously, as has been the case with so many other safety codes over human history.
It makes me really sad, embarrassed for our industry, and more disappointed in human nature.
I can confirm these exist, as growing up my father's phone number was on a some of those signs. Fancy systems would automatically call an answering service, but for smaller ones, he'd get a call (or page, given the era), usually from a neighbor or something.
It wasn't limited to rural areas either. I distinctly remember him taking service calls in ultra-wealthy neighborhoods on the weekends. One time, the gentleman who called in let my sister and I watch movies in his personal movie theater while my dad fixed the system.
The largest issues is that you must gather the data from sensors that can't interfere with the thing you are measuring, and that you must process it with computers that don't connect to the ones controlling the process. The first one is really just good engineering practice, and the second is already cheap and getting cheaper by the day.
Also, whatever you do at your process control, you should have some emergency overrides that set when the conditions get too abnormal. Those should be simple (AKA, no computers if possible) and stand-alone. Looks like they got this one right.
By running with loose tolerances and loose guarantees (and keeping systems as simple as possible) they remove the need for these tools - and their attack surface.
The action is stupid, but trespassing idiots get caught quickly - that's just "survival of the fittest" mechanics.
If you never trespassed in your life then you were probably not smart enough to get away with it?
As a kid I've tresspassed plenty, it often takes 'security' hours to even spot you
His comment says that small rural town water treatment plants don't have to run for 24/7. Not sure what you thought he was saying.
This sounds like it should be a punchline to one of those funny programming jokes https://news.ycombinator.com/item?id=25850739
There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,” and it seems like everyone just assumes that everyone else will do a perfect job implementing and configuring security. I fear the tune will only change after the first multi-million dollar lawsuit, and I hope all that costs is the money.
That's because the value proposition is only obvious when you substitute for "Stockholm" a city from one of the countries with cheaper labor.
In my limited experience with Industry 4.0, it smells like a combination of forcing a goldrush to sell shovels (so many players that want to be the platform which connects everything) on one side, and ongoing search to turn capex into opex on the other (that latter thing is a trend in pretty much all industries, though). I think there's enough companies that would happily replace their control systems (and control engineers) with prepackaged control-as-a-service which they don't have to know anything about, supplied by the lowest bidder, to which they can shift any liability if anything happens. This kind of setup does require remote access.
There are some packaged solutions but they all involve lots and lots of expert design, setup and management.
So quite often it actually is for the interaction with possibly expensive engineer from random place in the world who has specific knowledge for the system involved, as well as enabling remote operations when facilities are in less accessible location.
As for IIoT 4.0 - truth is a lot of industry was already heavily connected, and many functions I've seen so far are about getting deeper integration between ERP, MES, and individual work cells and workpiece tracking.
Even when the workpiece is fried chicken waiting to be put in a bun, or a cut of pipe that will next need to be appropriately cleaned, bent, welded, painted and finally become part of a ship assembly.
The answer is Yes. Very very yes. Especially when said programmer can't travel across borders due to Covid restrictions.
But even without Covid, it's a lot cheaper and time-effective to let people look at stuff and fix things from stockholm, or antwerp, or warsaw or whatnot. Else every time your robot sneezes, you have to book plane tickets and a hotel. But worst of all, you risk losing many hours of production due to travel time.
In contrast, with remote operation, you can log in, fix problems in well under 30 minutes, and Get Production Running Again.
In a situation where any kind of stoppage basically means the factory is Not Making Money, you can see the very strong value proposition here.
Putting a defenseless PLC or robot controller on the open internet is clearly not the best of plans.
(though the amount of people using teamviewer is telling)
Several ICS vendors like Tosibox and EWON make devices to accomplish this. I think Tosi has the more secure model, though I hate their proprietary dongles.
VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. You have to talk to someone in IT to get your files moved in/out.
I think Amazon went a different direction and uses Versiondog to monitor their automation systems and check for changes. I don't work there or know anyone on their automation team so I'm not aware of the details.
Still, I think you can have external access and be secure. You just need to balance things out with your business needs.
I support an environment almost exactly like this (albeit in a small manufacturing company). I don't love having one of the controls networks attached, in any way, to the LAN, but I understand the business requirements justify it.
It happens that there's a controls system running devices that could cause massive environmental impact in some malfunction scenarios. I am happy to report the plant, being held to account for things like public evacuation plans and hazmat filings with local first responders, has never asked about connecting that network to anything. That would be a walk-out-the-door type scenario for me. I worry that they'd just find somebody who wouldn't have those scruples, though.
If I may speak slightly out of turn to a stranger, using a possible & currently imagined future person less scrupulous does not modify in any way your obligation, however you perceive it, to act ethically.
Nobody was suggesting that it does. The person you're replying to specifically said the opposite.
I don't have those resources, nor do my Customers. I've got the various mix of Windows, Linux, and embedded devices that the Customer has purchased to serve their business applications. They (and I) don't have the clout or purchasing power to demand application vendors bend to our desires, so I'm left with making the best out of sub-optimal architecture, protocols, etc.
Google says, in the BeyondCorp III paper under the heading "Third-Party Software"[1]:
Third-party software has frequently proved troublesome, as sometimes it can’t present TLS certificates, and sometimes it assumes direct connectivity. In order to support these tools, we developed a solution to automatically establish encrypted point-to-point tunnels (using a TUN device). The software is unaware of the tunnel, and behaves as if it’s directly connected to the server.
So, they just do what I do and throw a VPN at it, albeit a client-to-server VPN serving an individual application rather than a client-to-network VPN like I might.
I do my best to segment the networks at my Customer sites, to use default-deny policies between security zones, to authenticate traffic flows to users and devices where possible, and when unable (because of limitations of client software/devices, usually) restrict access by source address. Within each security zone I try to make a worst-case assumption of an attacker getting complete access to the zone (compromising a host within the zone and getting arbitrary network access, for example) with things like private VLANs and host-based firewalls. I have to declare "bankruptcy" in some security zones (usually where there are embedded devices) where I have to rely only on network segmentation because the devices (or vendors) are too "stupid" to have host-based firewall functionality, authentication, encryption, etc. (These are the devices that fall over and die when they get port-scanned, yet somehow end up in mission-critical roles.)
I think the harsh reality is that, operating at the scale of small to mid-sized companies, IT and infosec are forced into a lot of bad places by vendors who don't care, and management who are focused on the bottom-line and who don't see security as anything other than something to purchase insurance for.
To put it another way: I have to make all this crap work. If I make it too difficult for the end users to work or for the vendors to support I'll be kicked to the curb and they'll find somebody else who will be less "difficult".
[1] https://storage.googleapis.com/pub-tools-public-publication-...
The risk of adding remote access to critical systems is the introduction of globally accessible single-point-of-failures. Given the nature of software, such an attack has an unlimited amount of time to be perfected before deployment and when finished can be deployed at effectively zero cost and complete in effectively zero time which provides no meaningful way to respond except with already deployed automated systems. So, the risk added with remote access is the risk of malicious catastrophic total system failure.
In this case, the water treatment facility treated the water for ~15,000 residents. In a similar case many years ago [1], a similar event occurred to a water treatment facility that treated the water for ~12,000 residents which resulted in 100 affected individuals before the effects were detected. So, we can reasonably assume that undetected water treatment tampering on a facility serving ~10,000 individuals will result in about ~100 affected individuals before the effects are detected. If there exists a way to tamper with a water treatment facility that would result in deaths for the affected individuals, which is quite likely, then that means the risk of remote access to the water treatment facility is ~100 deaths. So, as a society, we should ask the question: What is the standard of care that should be applied to a system where failure may result in the deaths of 100 people? And any business that wishes to add remote access to such a system must demonstrate to the satisfaction of society that they are taking that degree of care. It is not the role of society or the people to suffer for the convenience of business.
And in this case, I am certain that they are not taking an appropriate amount of care. The fact that you honestly suggested that an IT department would shove an AP in the ceiling for their convenience shows just how low our expectations are. In any other industry, such an act would be, in no uncertain terms, criminal negligence. That our standard assumption about the standard of care taken is criminal negligence shows just how far any of these companies is from actually deploying systems that have external access and have adequate security.
[1] https://www.spencerma.gov/sites/g/files/vyhlif1246/f/uploads...
Businesses need lower cost because they are under price pressure. Especially with small utilities. Remote access is one of those ways to lower their costs on personnel or vendor support.
There is still a whole lot of low hanging fruit in automation for improving security and access control. We're not going to get it from Rockwell for sure though.
Just so I am clear, doing what you say they are doing should be so unacceptable that it is not even viewed as an option. Anybody attempting to do so should incur costs so great that there would be no competitive advantage to offloading risk to society to the detriment of the people as the costs of doing so outweigh the benefits. If that prevents businesses from making certain profitable decisions due to the collateral damage they will cause then that seems like their problem.
So right now things the op posted are pretty much standard practice everywhere in most industries. I mostly work in EU, I have worked with construction companis, medical companies, hospitals and telcos, and practice like this is standard.
They will have some ungodly expensive security product that makes them change password ever 14 days, and makes intranet barely usable, but will have holes the size of the mountains in their infrastructure, because of this vendor or that cost savings etc.
When downtime is expensive, the pressure from the business is to err on the side of being able to get experts in to troubleshoot the system as easily as possible, vs guaranteeing that bad guys can't get in. The first they see all the time, and the second seems unreal until it actually happens...
[0] https://literature.rockwellautomation.com/idc/groups/literat...
They also have a satellite office over in Clearwater, Florida (which is trying to be like a little bay area copy, v2/3)
Interesting, but, Teamviewer also has been exploited and leaked creds, but took three years to confirm it: https://www.bleepingcomputer.com/news/security/teamviewer-co...
Or, if the client computer browsed a site, it'd actually start open an SMB share on the perps computer: https://www.bleepingcomputer.com/news/security/teamviewer-fi...
and a few other interesting vulnerabilities, hmm.
Teamviewer on a desktop, probably with a shared credential isn't very secure. Knowing this though, I doubt that it was a teamviewer exploit. My guess would be a disgruntled employee since they knew what to get into to change chemical set points.
I would be interested to know if the TeamViewer account in question had 2FA... probably not.
It's OK for a huge city operating many water treatment plants to decide that it is more efficient to automate and centralize and secure the network. It is horrendous that this is seen as the cheap solution for a small town.
What will stop the local city council be compliant on paper, ie them doing a tick box exercise and saying that their summer IT intern is the security department?
It would of course require significant political will to create these institutions and system of laws and regulations, but it could be similar in spirit to the kinds of controls the military has for software vendors that want to work with it.
Until the decision makers who demand the interconnection of these networks are held accountable it's isn't going to stop.
The cases I've seen have been to facilitate 24x7 off-site PLC vendor support access. I certainly see the business argument for the economics of off-site support for infrequently/improbably failure scenarios. At the very least, though, some type of physical interlock could have been employed (at the expense of some response time).
Edit: I think controls can exist to make this kind of situation tenable for at least some types of industrial controls applications. When you start getting to things like municipal water and power I start getting more antsy.
Honestly, I think this is a fine outcome. There is a dollar value per life. I don't think we're undervaluing the life yet.
Likewise these consultants are not just coming in and pointing their fingers at the obvious ICS on the internet. They are also providing services to understand why they were attached in the first place and where that process broke down, how to keep the current ease with which to operate the system, and implement the transitions.
In real life, the internals of a water plant are behind locked doors. Not everybody from Nairobi to Nantucket can get in and do as they like.
I'm afraid that trust in the public is definitely not the way to go with infrastructure and networked control systems.
I'm fairly sure the Iranian ultracentrifuges were not connected, and were hacked anyway. Stuxnet was complicated, but being disconnected is not a 100% protection.
We're fighting to keep these people from using unlicensed copies of TeamViewer for their primary access.
Nah, Dragos knows their shit. They'd be around even if ICS had good security.
The contractor and integrators then move on to the next project and copy what they did last time. Rinse , repeat.
We've been actively pressing for realistic security and access control planning in the contract stage, but that's slow going in and of itself and still only affects new or upgraded installations -- on facilities with an expected lifetime of 10-30 years.
Then look for a list of open modbus ports on the Internet and be wowed at all the industrial machinery that is just sat on the Internet...
They are a problem the way drunk driving is a problem.
You just don't ever do it. Ever.
No cyber security products are needed. No budget required.
These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist.