It’s worked well so far, but we do have an IT crew that would make most places jealous and IT is an area that is notoriously undervalued in the public sector.
Maybe there is a job where you do only the former, I just haven't found it yet.
There are a few Palo Alto and Cisco guys near me. They definitely aren’t answering the help desk phone but they are level 3 support. Same with my development team.
Not to mention large orgs have NOCs dedicated to this. The last company I was at at a couple CCAs in the NOC. They were smart but an odd breed.
Or they weren't able to detect exactly when the malware was deployed so they didn't know how far back the data corruption went, meaning they couldn't trust the backups even if they looked OK. One of the problems you face after any attack is trusting the system again. Verifying everything is correct is a very hard problem.
The interim IT manager had to setup a new website for the city w/ new email addresses?
They lost control of DNS and registration of their domain, it seems.
SCADA systems for water pumps were inoperable.
Online payments no longer functioned.
This seems like a very premeditated attack to me.
Its very common in custom hardware setups to have a standing system that interfaces with the physical hardware and PLC and then the user friendly software for instructing that controller on what you want to be on a PC talking to it over serial or the network. Obviously if the computer is inaccessible you can't adjust settings, but the system continues to run fine.
The AD server should be able to be destroyed without preventing the water supply from functioning.
I know of three small municipalities in my area (smaller than this town), and the utilities are not part of the flat network.
Not really. You can easily extract data and overwrite the disk surface.
If the malware manages to get itself running on the file server itself (and with root privileges), then sure. That's not a common case though.
As long as you harden it from crypto, there is no way for malware on client machines to force an overwrite of current backups.
I expect they are allowing their backup app to have read write access to manage cleaning up removing backups but just giving it write only access from the network would work too. And using a standalone server/app to manage the backups.
I'd still copy over a backup to a stand alone bucket not accessible by the network for something this critical.
Remember, when you're hiking and you're attacked by a bear, you don't need to outrun the bear. You only need to outrun the slowest hiker in your group.
I think that the last day’s work (or last hour’s work) will be lost or will require a lot of manual fixing regardless of whether they pay the ransom. If they pay, they’ll still have to fix partial database transactions, corrupt files, etc., for the attack date. If they don’t pay, they can recover from earlier good backups and reconstruct that one day’s worth. My reasoning is that the attack date’s data is going to be corrupt and untrustworthy in either case, and it’ll be equal work either way. (Or at least it’ll be less than $600,000 of work to fix that one day.)
I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.