News for hackers. Stuff that matters.
Re:Hacker increased chemical level at Oldsmar's city water s... by rsync on Monday February 08, 2021 @06:38PM (#26071889)
"Internet-accessble industrial control systems have been a problem for many years now ..."

They are a problem the way drunk driving is a problem.

You just don't ever do it. Ever.

No cyber security products are needed. No budget required.

These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist.

Re:Hacker increased chemical level at Oldsmar's city water s... by ac29 on Monday February 08, 2021 @07:19PM (#26072235)
In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks.

I regularly work with these sorts of water districts (larger, better funded ones as well). In reality, some of these small districts may only have 2 or 3 SCADA operators on staff. Sending them home with a pager, a tablet, a VPN password, and some overtime pay is a lot easier to get past the city council then taking on another two employees to cover the night shift for those rare events that need to be handled ASAP.

I could share some real horror stories, but it wouldnt be professionally appropriate. Suffice to say, this story did not surprise me at all.

Re:Hacker increased chemical level at Oldsmar's city water s... by rsync on Monday February 08, 2021 @11:03PM (#26073684)
"In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks."

I reject this line of thought.

A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades.

They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances.

As for telemetry ...

I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?

Re:Hacker increased chemical level at Oldsmar's city water s... by EvanAnderson on Tuesday February 09, 2021 @11:33AM (#26079441)
re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting.

I like your thinking and I try to espouse it myself (keeping things as simple as they can be-- keeping "technology" out of voting, not connecting things to networks that have no business being connected, etc). Short of a Battlestar Galactica-type "our machines rise up and try to kill us" event, though, I don't think the average person will ever understand the vulnerability inherent in networked computers or the risk/benefit tradeoff of connected vs. disconnected systems.

Even down at the level of local politics in a rural setting the "optics" of bringing technological solutions to bear on problems is seen as forward-thinking-- particularly when it "saves" the taxpayer money. I can't imagine trying to convince a local water board that moving away from a PLC-based system with a remote support vendor world fly, even citing this example.

This event will be another opportunity for more security vendors to cite in case studies justifying their products. More layers of garbage will build up on a foundation of protocols and design philosophies that grew up in an era of disconnected systems with lower stakes and a less complex threat model.

I don't see big money to be made in providing sensible levels of connectivity and security to this kind of infrastructure. I don't see industry stepping-up because of that. Maybe regulation is the answer, though I'd just expect regulatory capture to take over, and have it become another "PCI". Maybe a lot of people have to die before society takes it seriously, as has been the case with so many other safety codes over human history.

It makes me really sad, embarrassed for our industry, and more disappointed in human nature.

Re:Hacker increased chemical level at Oldsmar's city water s... by mywittyname on Tuesday February 09, 2021 @09:31PM (#26085146)
> re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting.

I can confirm these exist, as growing up my father's phone number was on a some of those signs. Fancy systems would automatically call an answering service, but for smaller ones, he'd get a call (or page, given the era), usually from a neighbor or something.

It wasn't limited to rural areas either. I distinctly remember him taking service calls in ultra-wealthy neighborhoods on the weekends. One time, the gentleman who called in let my sister and I watch movies in his personal movie theater while my dad fixed the system.

Re:Hacker increased chemical level at Oldsmar's city water s... by marcosdumay on Tuesday February 09, 2021 @08:58AM (#26077458)
There is little problem on pushing telemetry into the internet.

The largest issues is that you must gather the data from sensors that can't interfere with the thing you are measuring, and that you must process it with computers that don't connect to the ones controlling the process. The first one is really just good engineering practice, and the second is already cheap and getting cheaper by the day.

Also, whatever you do at your process control, you should have some emergency overrides that set when the conditions get too abnormal. Those should be simple (AKA, no computers if possible) and stand-alone. Looks like they got this one right.

Re:Hacker increased chemical level at Oldsmar's city water s... by discreteevent on Tuesday February 09, 2021 @10:00AM (#26078221)
Plus the benefit of setting something up to get telemetry as you described, is that someone won't later be tempted to use teamviewer (as in the article example) to open up the whole control system just to view some of that telemetery.
Re:Hacker increased chemical level at Oldsmar's city water s... by sillysaurusx on Monday February 08, 2021 @11:54PM (#26074030)
I invite you to become a pentester for a year and see if you still reject this line of thought.
Re:Hacker increased chemical level at Oldsmar's city water s... by rsync on Tuesday February 09, 2021 @01:27AM (#26074516)
You misunderstand - I am saying they need neither the computers nor the networks.

By running with loose tolerances and loose guarantees (and keeping systems as simple as possible) they remove the need for these tools - and their attack surface.

Re:Hacker increased chemical level at Oldsmar's city water s... by josefx on Tuesday February 09, 2021 @01:54AM (#26074652)
pen testing often involves walking into a building looking like you should be there. If no one is around to let you in it also can involve half a minute of getting the door open with some primitive every day tools. No computers or software required.
Re:Hacker increased chemical level at Oldsmar's city water s... by consp on Tuesday February 09, 2021 @03:05AM (#26074991)
You just switched the attacker model from "script kiddie somewhere in the world, changing stuff for the fun of it" to "physical presence with specific malicious intent". Those are not in the same ballpark, not even the same country.
Re:Hacker increased chemical level at Oldsmar's city water s... by josefx on Tuesday February 09, 2021 @04:04AM (#26075334)
You underestimate the stupid things local kids can get up to for fun if they thought they wouldn't get caught. So you can at least throw out the "specific malicious intent".
Re:Hacker increased chemical level at Oldsmar's city water s... by elbear on Tuesday February 09, 2021 @04:42AM (#26075570)
Ok, but how many stupid local kids are there and how many worldwide script kiddies?
Re:Hacker increased chemical level at Oldsmar's city water s... by heavenlyblue on Tuesday February 09, 2021 @05:52AM (#26075980)
Kids who are trespassing are not stupid. You have to take the risk, observe the behaviour of the security and then behave reasonably so that people would forgive you if you get caught.

The action is stupid, but trespassing idiots get caught quickly - that's just "survival of the fittest" mechanics.

If you never trespassed in your life then you were probably not smart enough to get away with it?

Re:Hacker increased chemical level at Oldsmar's city water s... by ClumsyPilot on Tuesday February 09, 2021 @06:29AM (#26076246)
Security is a big word, we are talking about large facilities with like 1 or 2 people on site.

As a kid I've tresspassed plenty, it often takes 'security' hours to even spot you

Re:Hacker increased chemical level at Oldsmar's city water s... by codyb on Tuesday February 09, 2021 @07:17PM (#26084340)
I’m not particularly worried about a group of kids contaminating a single rural water supply, I’m more worried about coordinated effort by a malicious nation state to accumulate security cracks and then use them in a coordinated effort during a war time scenario.
Re:Hacker increased chemical level at Oldsmar's city water s... by mgreenleaf on Tuesday February 09, 2021 @08:45AM (#26077341)
I think it would be quite possible to have the control systems completely offline, while installing a reputable alarm system that is connected to the internet. If those two things have no network connection, then you could monitor the premises, but even a remote hacking of that system would not enable changing of chemical levels.
Re:Hacker increased chemical level at Oldsmar's city water s... by chopin on Wednesday February 10, 2021 @09:04AM (#26089228)
But you have this problem as well, even if you use networked systems. Now you need to mitigate both.
Re:Hacker increased chemical level at Oldsmar's city water s... by philtar on Tuesday February 09, 2021 @02:49AM (#26074912)
You must be a pentester, because the commenter above you wasn't talking about pentesting at all.

His comment says that small rural town water treatment plants don't have to run for 24/7. Not sure what you thought he was saying.

Re:Hacker increased chemical level at Oldsmar's city water s... by bryanrasmussen on Tuesday February 09, 2021 @03:42AM (#26075217)
>You must be a pentester, because the commenter above you wasn't talking about pentesting at all.

This sounds like it should be a punchline to one of those funny programming jokes https://news.ycombinator.com/item?id=25850739

Re:Hacker increased chemical level at Oldsmar's city water s... by ClumsyPilot on Tuesday February 09, 2021 @06:39AM (#26076324)
It would make waay more sence for the entire state / country to buy software for their water systems all at once and set up a department of a few dozen people that travel the country and make sure its setup properly. Why would you not have all water treatment in the country on the same software platform?
Re:Hacker increased chemical level at Oldsmar's city water s... by croon on Tuesday February 09, 2021 @07:10AM (#26076572)
A lot of these problems with inefficiency comes down to "muh state's rights", and less federal involvement. I think my stance shines through, but I understand it's debatable per case.
Re:Hacker increased chemical level at Oldsmar's city water s... by joe_the_user on Monday February 08, 2021 @10:51PM (#26073591)
You parent is saying: "just never put your stuff in the Internet". I suppose it really is "that hard" to tell clients this and we can expect more problem down the line.
Re:Hacker increased chemical level at Oldsmar's city water s... by tsimionescu on Tuesday February 09, 2021 @02:32AM (#26074825)
How did they operate before computing became a mainstream thing? Utilities as a comcept are significantly older than computing itself, and certainly mainstream, internet-connected computation.
Re:Hacker increased chemical level at Oldsmar's city water s... by wnkrshm on Tuesday February 09, 2021 @03:56AM (#26075291)
They had to pay more for employees who could be on site at unreasonable hours.
Re:Hacker increased chemical level at Oldsmar's city water s... by tsimionescu on Tuesday February 09, 2021 @05:01AM (#26075693)
Perfect, unemployment is a major problem because of Covid19 anyway.
Re:Hacker increased chemical level at Oldsmar's city water s... by redbeard0x0a on Tuesday February 09, 2021 @11:02AM (#26079033)
COVID19 is also decimating budgets
Re:Hacker increased chemical level at Oldsmar's city water s... by windsurfer on Tuesday February 09, 2021 @12:01AM (#26074075)
If all it takes to prevent the poisoning of an entire city's water supply is two employees, I certainly hope my governments are choosing to hire those two employees.
Re:Hacker increased chemical level at Oldsmar's city water s... by o-__-o on Tuesday February 09, 2021 @12:56AM (#26074345)
I will make an assumption here that you don’t live in a rural city...
Re:Hacker increased chemical level at Oldsmar's city water s... by paledot on Tuesday February 09, 2021 @01:05AM (#26074393)
Or, let's face it, in the United States.
Re:Hacker increased chemical level at Oldsmar's city water s... by nefitty on Monday February 08, 2021 @09:49PM (#26073318)
Of all the commentary I’ve read on this issue so far, this might be the scariest anecdote so far.
Re:Hacker increased chemical level at Oldsmar's city water s... by crystalmeph on Monday February 08, 2021 @07:48PM (#26072472)
I work in industrial automation, and I agree. There’s constant rhetoric about buzzwords like “Industry 4.0,” which, if it means anything specifically, means “connect all the things.”

There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,” and it seems like everyone just assumes that everyone else will do a perfect job implementing and configuring security. I fear the tune will only change after the first multi-million dollar lawsuit, and I hope all that costs is the money.

Re:Hacker increased chemical level at Oldsmar's city water s... by TeMPOraL on Tuesday February 09, 2021 @02:43AM (#26074889)
> There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,”

That's because the value proposition is only obvious when you substitute for "Stockholm" a city from one of the countries with cheaper labor.

In my limited experience with Industry 4.0, it smells like a combination of forcing a goldrush to sell shovels (so many players that want to be the platform which connects everything) on one side, and ongoing search to turn capex into opex on the other (that latter thing is a trend in pretty much all industries, though). I think there's enough companies that would happily replace their control systems (and control engineers) with prepackaged control-as-a-service which they don't have to know anything about, supplied by the lowest bidder, to which they can shift any liability if anything happens. This kind of setup does require remote access.

Re:Hacker increased chemical level at Oldsmar's city water s... by p_l on Tuesday February 09, 2021 @09:25AM (#26077801)
The thing is, there is no "one size fits all" option most of the time.

There are some packaged solutions but they all involve lots and lots of expert design, setup and management.

So quite often it actually is for the interaction with possibly expensive engineer from random place in the world who has specific knowledge for the system involved, as well as enabling remote operations when facilities are in less accessible location.

As for IIoT 4.0 - truth is a lot of industry was already heavily connected, and many functions I've seen so far are about getting deeper integration between ERP, MES, and individual work cells and workpiece tracking.

Even when the workpiece is fried chicken waiting to be put in a bun, or a cut of pipe that will next need to be appropriately cleaned, bent, welded, painted and finally become part of a ship assembly.

Re:Hacker increased chemical level at Oldsmar's city water s... by wu_187 on Tuesday February 09, 2021 @01:51PM (#26081158)
You hit the nail on the head. Once (legally) viable, I guarantee a large portion of our public sector infrastructure maintenance will be outsourced to the lowest bidder in the guise of saving tax money. All without any due diligence as to the safety ramifications of said actions.
Re:Hacker increased chemical level at Oldsmar's city water s... by Kim_Bruning on Tuesday February 09, 2021 @08:43AM (#26077312)
> “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,”

The answer is Yes. Very very yes. Especially when said programmer can't travel across borders due to Covid restrictions.

But even without Covid, it's a lot cheaper and time-effective to let people look at stuff and fix things from stockholm, or antwerp, or warsaw or whatnot. Else every time your robot sneezes, you have to book plane tickets and a hotel. But worst of all, you risk losing many hours of production due to travel time.

In contrast, with remote operation, you can log in, fix problems in well under 30 minutes, and Get Production Running Again.

In a situation where any kind of stoppage basically means the factory is Not Making Money, you can see the very strong value proposition here.

Re:Hacker increased chemical level at Oldsmar's city water s... by kritiko on Tuesday February 09, 2021 @10:55AM (#26078939)
I don't think it's an all-or-nothing proposition. You can share telemetry and get patches from a remote team without having the equipment connected to the internet to reduce the risk of destroying expensive equipment.
Re:Hacker increased chemical level at Oldsmar's city water s... by Kim_Bruning on Tuesday February 09, 2021 @11:17AM (#26079254)
VPN or jump host, sufficiently firewalled in all directions.

Putting a defenseless PLC or robot controller on the open internet is clearly not the best of plans.

(though the amount of people using teamviewer is telling)

Re:Hacker increased chemical level at Oldsmar's city water s... by karmicthreat on Monday February 08, 2021 @06:54PM (#26072030)
I think these days it is becoming a business need though. These systems are made by vendors who probably need remote access. Also if the plant has a relatively unsophisticated IT department then someone is just going to shove an AP in the ceiling so they can check things when they get called at 1AM.

Several ICS vendors like Tosibox and EWON make devices to accomplish this. I think Tosi has the more secure model, though I hate their proprietary dongles.

VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. You have to talk to someone in IT to get your files moved in/out.

I think Amazon went a different direction and uses Versiondog to monitor their automation systems and check for changes. I don't work there or know anyone on their automation team so I'm not aware of the details.

Still, I think you can have external access and be secure. You just need to balance things out with your business needs.

Re:Hacker increased chemical level at Oldsmar's city water s... by EvanAnderson on Monday February 08, 2021 @07:07PM (#26072128)
VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on.

I support an environment almost exactly like this (albeit in a small manufacturing company). I don't love having one of the controls networks attached, in any way, to the LAN, but I understand the business requirements justify it.

It happens that there's a controls system running devices that could cause massive environmental impact in some malfunction scenarios. I am happy to report the plant, being held to account for things like public evacuation plans and hazmat filings with local first responders, has never asked about connecting that network to anything. That would be a walk-out-the-door type scenario for me. I worry that they'd just find somebody who wouldn't have those scruples, though.

Re:Hacker increased chemical level at Oldsmar's city water s... by just_steve_h on Monday February 08, 2021 @07:17PM (#26072216)
> I worry that they'd just find somebody who wouldn't have those scruples, though.

If I may speak slightly out of turn to a stranger, using a possible & currently imagined future person less scrupulous does not modify in any way your obligation, however you perceive it, to act ethically.

Re:Hacker increased chemical level at Oldsmar's city water s... by EvanAnderson on Monday February 08, 2021 @07:42PM (#26072421)
I've warned them about my concerns. If I stop working with them, and I have no further knowledge of thir situation, I don't see what else there would be for me to do.
Re:Hacker increased chemical level at Oldsmar's city water s... by Dylan16807 on Tuesday February 09, 2021 @01:31AM (#26074551)
Ohhhhkay?

Nobody was suggesting that it does. The person you're replying to specifically said the opposite.

Re:Hacker increased chemical level at Oldsmar's city water s... by uzakov on Tuesday February 09, 2021 @03:25AM (#26075109)
What are your thoughts on something like this in your line of work? https://cloud.google.com/beyondcorp
Re:Hacker increased chemical level at Oldsmar's city water s... by EvanAnderson on Tuesday February 09, 2021 @10:21AM (#26078450)
I may be too much of a simple "IT guy" to grok the deep meaning of BeyondCorp. I read thru some of the various papers when they came out and always came back to the thought "Yeah, that's nice if you have the resources to exert control over that much of your technology stack."

I don't have those resources, nor do my Customers. I've got the various mix of Windows, Linux, and embedded devices that the Customer has purchased to serve their business applications. They (and I) don't have the clout or purchasing power to demand application vendors bend to our desires, so I'm left with making the best out of sub-optimal architecture, protocols, etc.

Google says, in the BeyondCorp III paper under the heading "Third-Party Software"[1]:

Third-party software has frequently proved troublesome, as sometimes it can’t present TLS certificates, and sometimes it assumes direct connectivity. In order to support these tools, we developed a solution to automatically establish encrypted point-to-point tunnels (using a TUN device). The software is unaware of the tunnel, and behaves as if it’s directly connected to the server.

So, they just do what I do and throw a VPN at it, albeit a client-to-server VPN serving an individual application rather than a client-to-network VPN like I might.

I do my best to segment the networks at my Customer sites, to use default-deny policies between security zones, to authenticate traffic flows to users and devices where possible, and when unable (because of limitations of client software/devices, usually) restrict access by source address. Within each security zone I try to make a worst-case assumption of an attacker getting complete access to the zone (compromising a host within the zone and getting arbitrary network access, for example) with things like private VLANs and host-based firewalls. I have to declare "bankruptcy" in some security zones (usually where there are embedded devices) where I have to rely only on network segmentation because the devices (or vendors) are too "stupid" to have host-based firewall functionality, authentication, encryption, etc. (These are the devices that fall over and die when they get port-scanned, yet somehow end up in mission-critical roles.)

I think the harsh reality is that, operating at the scale of small to mid-sized companies, IT and infosec are forced into a lot of bad places by vendors who don't care, and management who are focused on the bottom-line and who don't see security as anything other than something to purchase insurance for.

To put it another way: I have to make all this crap work. If I make it too difficult for the end users to work or for the vendors to support I'll be kicked to the curb and they'll find somebody else who will be less "difficult".

[1] https://storage.googleapis.com/pub-tools-public-publication-...

Re:Hacker increased chemical level at Oldsmar's city water s... by karmicthreat on Monday February 08, 2021 @07:24PM (#26072280)
That's an understandable scenario and good on you guys for balancing the risks.
Re:Hacker increased chemical level at Oldsmar's city water s... by Veserv on Monday February 08, 2021 @08:16PM (#26072661)
It is not a business "need". These systems have functioned without remote access perfectly well for decades. It is a business "want" and thus must be balanced against any new risks relative to historical risks.

The risk of adding remote access to critical systems is the introduction of globally accessible single-point-of-failures. Given the nature of software, such an attack has an unlimited amount of time to be perfected before deployment and when finished can be deployed at effectively zero cost and complete in effectively zero time which provides no meaningful way to respond except with already deployed automated systems. So, the risk added with remote access is the risk of malicious catastrophic total system failure.

In this case, the water treatment facility treated the water for ~15,000 residents. In a similar case many years ago [1], a similar event occurred to a water treatment facility that treated the water for ~12,000 residents which resulted in 100 affected individuals before the effects were detected. So, we can reasonably assume that undetected water treatment tampering on a facility serving ~10,000 individuals will result in about ~100 affected individuals before the effects are detected. If there exists a way to tamper with a water treatment facility that would result in deaths for the affected individuals, which is quite likely, then that means the risk of remote access to the water treatment facility is ~100 deaths. So, as a society, we should ask the question: What is the standard of care that should be applied to a system where failure may result in the deaths of 100 people? And any business that wishes to add remote access to such a system must demonstrate to the satisfaction of society that they are taking that degree of care. It is not the role of society or the people to suffer for the convenience of business.

And in this case, I am certain that they are not taking an appropriate amount of care. The fact that you honestly suggested that an IT department would shove an AP in the ceiling for their convenience shows just how low our expectations are. In any other industry, such an act would be, in no uncertain terms, criminal negligence. That our standard assumption about the standard of care taken is criminal negligence shows just how far any of these companies is from actually deploying systems that have external access and have adequate security.

[1] https://www.spencerma.gov/sites/g/files/vyhlif1246/f/uploads...

Re:Hacker increased chemical level at Oldsmar's city water s... by karmicthreat on Monday February 08, 2021 @10:13PM (#26073423)
Oh, you misunderstand. IT is an impediment to many control engineers. It's the automation techs and engineers that will work around the IT department if IT can't supply solutions. One of the more common ones being hide an AP or like in the article, use teamviewer or other remote access software. Then just share a common credential because nobody wants to actually pay for teamviewer.

Businesses need lower cost because they are under price pressure. Especially with small utilities. Remote access is one of those ways to lower their costs on personnel or vendor support.

There is still a whole lot of low hanging fruit in automation for improving security and access control. We're not going to get it from Rockwell for sure though.

Re:Hacker increased chemical level at Oldsmar's city water s... by Veserv on Monday February 08, 2021 @11:58PM (#26074056)
I understood perfectly. I am just saying that such actions should be criminal and any reasonable lay person who was properly made aware of what is occurring would agree. Lowering costs is no excuse for engaging in criminal negligence and any tradeoff that has an outcome that would qualify as criminal negligence is socially unacceptable. That is not a proper balancing of business needs, that is pawning off immense risk to society for the convenience of a business.

Just so I am clear, doing what you say they are doing should be so unacceptable that it is not even viewed as an option. Anybody attempting to do so should incur costs so great that there would be no competitive advantage to offloading risk to society to the detriment of the people as the costs of doing so outweigh the benefits. If that prevents businesses from making certain profitable decisions due to the collateral damage they will cause then that seems like their problem.

Re:Hacker increased chemical level at Oldsmar's city water s... by unionpivo on Tuesday February 09, 2021 @05:28AM (#26075846)
Maybe we will get there someday, but we are not even close to that right now. Hell we are not even in same galaxy.

So right now things the op posted are pretty much standard practice everywhere in most industries. I mostly work in EU, I have worked with construction companis, medical companies, hospitals and telcos, and practice like this is standard.

They will have some ungodly expensive security product that makes them change password ever 14 days, and makes intranet barely usable, but will have holes the size of the mountains in their infrastructure, because of this vendor or that cost savings etc.

Re:Hacker increased chemical level at Oldsmar's city water s... by DrPhish on Wednesday February 10, 2021 @04:06PM (#26094799)
Rockwell definitely has some questionable security on individual products, but they partnered with Cisco for their Converged Plantwide Ethernet Design [0] which is actually pretty well thought out, and if implemented properly covers off most of the biggest risks. The problem is either that people don't know about it, don't bother to read it, or can't get organizational buy-in to implement it.

When downtime is expensive, the pressure from the business is to err on the side of being able to get experts in to troubleshoot the system as easily as possible, vs guaranteeing that bad guys can't get in. The first they see all the time, and the second seems unreal until it actually happens...

[0] https://literature.rockwellautomation.com/idc/groups/literat...

Re:Hacker increased chemical level at Oldsmar's city water s... by closeparen on Monday February 08, 2021 @07:34PM (#26072369)
Remote Desktop was exactly the mechanism here, the attacker used TeamViewer to work the UI on a plant operator’s desktop and he happened to be watching.
Re:Hacker increased chemical level at Oldsmar's city water s... by rootsudo on Monday February 08, 2021 @07:58PM (#26072547)
Seeing reports now that it was indeed Teamviewer!

They also have a satellite office over in Clearwater, Florida (which is trying to be like a little bay area copy, v2/3)

Interesting, but, Teamviewer also has been exploited and leaked creds, but took three years to confirm it: https://www.bleepingcomputer.com/news/security/teamviewer-co...

Or, if the client computer browsed a site, it'd actually start open an SMB share on the perps computer: https://www.bleepingcomputer.com/news/security/teamviewer-fi...

and a few other interesting vulnerabilities, hmm.

Re:Hacker increased chemical level at Oldsmar's city water s... by karmicthreat on Monday February 08, 2021 @08:06PM (#26072600)
Citrix is a little more than teamviewer. And can be encapsulated in a VPN as well.

Teamviewer on a desktop, probably with a shared credential isn't very secure. Knowing this though, I doubt that it was a teamviewer exploit. My guess would be a disgruntled employee since they knew what to get into to change chemical set points.

Re:Hacker increased chemical level at Oldsmar's city water s... by closeparen on Monday February 08, 2021 @08:57PM (#26072926)
They both expose Remote Desktop to the internet given the proper credentials, and I’m guessing here but I think it’s pretty likely that the attacker had credentials. Whether it was a disgruntled insider, a dumb password, or (most likely) a reused password from a leak somewhere.

I would be interested to know if the TeamViewer account in question had 2FA... probably not.

Re:Hacker increased chemical level at Oldsmar's city water s... by tsimionescu on Tuesday February 09, 2021 @02:40AM (#26074863)
Ideally, we should align our incentives such that having na internet-connected automation system is far more expensive than having one disconnected from the network. You should be forced by law to have a certain number of security experts on-call for any such system, periodic audits and pen-tests on your own expense etc.

It's OK for a huge city operating many water treatment plants to decide that it is more efficient to automate and centralize and secure the network. It is horrendous that this is seen as the cheap solution for a small town.

Re:Hacker increased chemical level at Oldsmar's city water s... by uzakov on Tuesday February 09, 2021 @03:24AM (#26075098)
I agree with your comment but want to ask a couple of questions to see how you see it working it practice:

What will stop the local city council be compliant on paper, ie them doing a tick box exercise and saying that their summer IT intern is the security department?

Re:Hacker increased chemical level at Oldsmar's city water s... by tsimionescu on Tuesday February 09, 2021 @05:00AM (#26075685)
I'm not a policy design expert by any means, and it's not like I've given this thorough thought. I expect some amount of red tape and controls from a government agency would be the proper way to enforce it.

It would of course require significant political will to create these institutions and system of laws and regulations, but it could be similar in spirit to the kinds of controls the military has for software vendors that want to work with it.

Re:Hacker increased chemical level at Oldsmar's city water s... by EvanAnderson on Monday February 08, 2021 @06:56PM (#26072045)
Yeah, but it's sadly common. I have personal experience with two such situations in my work over the last 16 years and I'm just some two-bit general IT contractor in Ohio, US.

Until the decision makers who demand the interconnection of these networks are held accountable it's isn't going to stop.

The cases I've seen have been to facilitate 24x7 off-site PLC vendor support access. I certainly see the business argument for the economics of off-site support for infrequently/improbably failure scenarios. At the very least, though, some type of physical interlock could have been employed (at the expense of some response time).

Edit: I think controls can exist to make this kind of situation tenable for at least some types of industrial controls applications. When you start getting to things like municipal water and power I start getting more antsy.

Re:Hacker increased chemical level at Oldsmar's city water s... by jacquesm on Tuesday February 09, 2021 @01:50AM (#26074640)
Totally agreed. The reason why these systems are network connected is to save a few pennies on periodical drive-by's but they open up a whole can of worms in terms of risk that those entities are very ill equipped to deal with. The same was happening with SCADA systems for building management. Systems that were quite literally wide open were given an IPV4 address based on the assumption that since all they did was run HVAC controllers on obscure UDP ports that they were safe and nobody would bother with them.
Re:Hacker increased chemical level at Oldsmar's city water s... by renewiltord on Monday February 08, 2021 @07:23PM (#26072272)
Sure, now you're paying 4x the price because you need two more operators so you can staff this place 100% of the time. And it's not even this guy. He probably reacts to like 4 different plants. Now you need like each of those places to have this guy in driving distance. You're not going to make it.

Honestly, I think this is a fine outcome. There is a dollar value per life. I don't think we're undervaluing the life yet.

Re:Hacker increased chemical level at Oldsmar's city water s... by johnwyles on Tuesday February 09, 2021 @07:36PM (#26084466)
Odd to pick HRBlock/Intuit as they are used for convenience and because we don't have a central system like Sweden and have in the US 50 different states and territories with 50+ different tax rules and so many edge cases in existence it's almost the rule than the exception each individual has one.

Likewise these consultants are not just coming in and pointing their fingers at the obvious ICS on the internet. They are also providing services to understand why they were attached in the first place and where that process broke down, how to keep the current ease with which to operate the system, and implement the transitions.

Re:Hacker increased chemical level at Oldsmar's city water s... by JoeAltmaier on Tuesday February 09, 2021 @08:32AM (#26077206)
Yet on the internet there are astronomical levels of 'griefers', people who just want to see the world burn. The internet magnifies this tremendously. The internet offers the appearance of anonymity. This is a dangerous combination. Drunk drivers are in the accident too. They take damage. Not so for a hack.

In real life, the internals of a water plant are behind locked doors. Not everybody from Nairobi to Nantucket can get in and do as they like.

I'm afraid that trust in the public is definitely not the way to go with infrastructure and networked control systems.

Re:Hacker increased chemical level at Oldsmar's city water s... by tgv on Tuesday February 09, 2021 @03:14AM (#26075034)
> No cyber security products are needed.

I'm fairly sure the Iranian ultracentrifuges were not connected, and were hacked anyway. Stuxnet was complicated, but being disconnected is not a 100% protection.

Re:Hacker increased chemical level at Oldsmar's city water s... by rob74 on Tuesday February 09, 2021 @03:20AM (#26075075)
Not 100%, but it takes orders of magnitude more motivation (and a nuclear program that would threaten your country if it were successful undoubtedly provides that) to accomplish...
Re:Hacker increased chemical level at Oldsmar's city water s... by tgv on Tuesday February 09, 2021 @07:24AM (#26076663)
Sure, but apart from low to medium effort hacking, cyber-warfare is still a possibility. Disabling infrastructure would be a high priority.
Re:Hacker increased chemical level at Oldsmar's city water s... by uzakov on Tuesday February 09, 2021 @03:22AM (#26075082)
That is a slightly different Threat Model though, as well as(though I could be wrong on this one) the capabilities of attackers.
Re:Hacker increased chemical level at Oldsmar's city water s... by netflixandkill on Tuesday February 09, 2021 @02:26AM (#26074796)
Except everyone already does it and that toothpaste is never going back in the tube.

We're fighting to keep these people from using unlicensed copies of TeamViewer for their primary access.

Re:Hacker increased chemical level at Oldsmar's city water s... by vsareto on Monday February 08, 2021 @09:19PM (#26073113)
>These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist.

Nah, Dragos knows their shit. They'd be around even if ICS had good security.

Re:Hacker increased chemical level at Oldsmar's city water s... by netflixandkill on Tuesday February 09, 2021 @03:42AM (#26075213)
We work with Dragos fairly regularly, they're solid. The main problem is that people who even consider the security or integrity of these systems are brought in years after they were specced, built, and more or less abandoned as built.

The contractor and integrators then move on to the next project and copy what they did last time. Rinse , repeat.

We've been actively pressing for realistic security and access control planning in the contract stage, but that's slow going in and of itself and still only affects new or upgraded installations -- on facilities with an expected lifetime of 10-30 years.

Re:Hacker increased chemical level at Oldsmar's city water s... by 72deluxe on Tuesday February 09, 2021 @06:06AM (#26076095)
Open your modbus port on your server and see how often you get hit!

Then look for a list of open modbus ports on the Internet and be wowed at all the industrial machinery that is just sat on the Internet...

Re:Hacker increased chemical level at Oldsmar's city water s... by rootw0rm on Tuesday February 09, 2021 @05:37AM (#26075903)
best comment i've read in a while.

KISS