No. $18 million was an estimate somebody gave once, who knows where it came from.

In fact, damages have not been repaired in Baltimore. 6 weeks later, most city services are still down. You can't pay a parking ticket or a water bill online. (You can send a check in; I am not sure where they record that you paid when they cash your check, and am not particularly confident they'll actually have a record I paid).

We in fact do not know how much money they've spent thus far, there have been no press briefings on this. Estimates of how much they will spend before it's over (will it ever be over?)... we all know how IT estimates work.

I think it will probably be quite a bit more than $18 million. And then there's estimating "damage to the economy." (There were two weeks when real estate transfers were frozen, because there was no way to check city liens. They can be done now, using a paper-based system that actually has those involved in the transaction sign an unusual contract agreeing to take on liability for unknown liens in unusual ways (I'm being vague cause I don't totally understand it), that some but not all title companies are willing to use).

The Baltimore ransomers only wanted ~$100K. If I were the mayor, yeah I'd pay it.

</Baltimore resident>

If you're running a bunch of critical services you need to secure them, and that takes millions every year, it's unavoidable. A single hack that exposes this is irrelevant

I don't know how much of the money/time that will be spent (the "$18 million" figure is entirely imaginary) to instead try to recover their data/systems to functional... but I don't think it's a great use of money.

But indeed, actually running secure systems is what they got to be focusing on, and will be expensive. I think many organizations are going to find that they can't actually afford to do what's needed to reliably run the systems they already rely on, having not been running them reliably or securely.

I hope that the 'recovery' efforts, done in an emergency fashion, don't distract them from getting to that point and figuring out what they're going to do about it.

I'm not saying that Baltimore is handling the situation correctly but there would be costs well above the x hundred thousand dollars the ransomers would be asking for either way.

Some of the crypto-blackmailers use public bitcoin addresses, and they don't get as much money as you'd expect.

For the fact you'd probably have to move your life to Russia to escape the FBI, it doesn't pay very well.

I think that the last day’s work (or last hour’s work) will be lost or will require a lot of manual fixing regardless of whether they pay the ransom. If they pay, they’ll still have to fix partial database transactions, corrupt files, etc., for the attack date. If they don’t pay, they can recover from earlier good backups and reconstruct that one day’s worth. My reasoning is that the attack date’s data is going to be corrupt and untrustworthy in either case, and it’ll be equal work either way. (Or at least it’ll be less than $600,000 of work to fix that one day.)

I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.

It’s worked well so far, but we do have an IT crew that would make most places jealous and IT is an area that is notoriously undervalued in the public sector.

Maybe there is a job where you do only the former, I just haven't found it yet.

There are a few Palo Alto and Cisco guys near me. They definitely aren’t answering the help desk phone but they are level 3 support. Same with my development team.

Not to mention large orgs have NOCs dedicated to this. The last company I was at at a couple CCAs in the NOC. They were smart but an odd breed.

Or they weren't able to detect exactly when the malware was deployed so they didn't know how far back the data corruption went, meaning they couldn't trust the backups even if they looked OK. One of the problems you face after any attack is trusting the system again. Verifying everything is correct is a very hard problem.

The interim IT manager had to setup a new website for the city w/ new email addresses?

They lost control of DNS and registration of their domain, it seems.

SCADA systems for water pumps were inoperable.

Online payments no longer functioned.

This seems like a very premeditated attack to me.

Its very common in custom hardware setups to have a standing system that interfaces with the physical hardware and PLC and then the user friendly software for instructing that controller on what you want to be on a PC talking to it over serial or the network. Obviously if the computer is inaccessible you can't adjust settings, but the system continues to run fine.

The AD server should be able to be destroyed without preventing the water supply from functioning.

I know of three small municipalities in my area (smaller than this town), and the utilities are not part of the flat network.

Not really. You can easily extract data and overwrite the disk surface.

If the malware manages to get itself running on the file server itself (and with root privileges), then sure. That's not a common case though.

As long as you harden it from crypto, there is no way for malware on client machines to force an overwrite of current backups.

I expect they are allowing their backup app to have read write access to manage cleaning up removing backups but just giving it write only access from the network would work too. And using a standalone server/app to manage the backups.

I'd still copy over a backup to a stand alone bucket not accessible by the network for something this critical.

Remember, when you're hiking and you're attacked by a bear, you don't need to outrun the bear. You only need to outrun the slowest hiker in your group.

>Underscoring the enormity of the city’s troubles, Mr. Williams explained that the webmaster hoped to get to that soon.

>“He’s been working very feverishly to get that done,” Mr. Williams said.

...the webmaster is working feverishly to post a static piece of text to a website? I guess it really is hard to fire government workers.

Add to that all sorts of other things that might be getting in the way and you can have someone working feverishly to get it done.

The problem is that the credentials that the webmaster needs likely only exist in an Excel spreadsheet that was saved on the desktop of the "Administrator" user account on one of the machines that got hit.

I wanna know what happened to the DNS records.

But the problem is that those that don't pay hurt even more.

The US makes it illegal to pay kidnapper ransom and, as a result, US citizens have much worse outcomes (often murdered) when they are ransomed abroad.

The theory that it makes US citizens less attractive targets is confounded by the fact that some families/friends of the victims can and do pay anyway (illegally).

Planet money did a podcast on it.

https://www.npr.org/sections/money/2017/09/01/548032302/epis...

Goes into the details of kidnapping as a business venture from the kidnappers perspective, and how a price equilibrium is found between 'buyers' (ransom payers) and 'sellers' (kidnappers).

On interesting story was when the partner of a small business owner got kidnapped as punishment for failing to pay some protection money. When the business owner went to negotiate the ransom, the kidnappers had had an accountant already go through the businesses fiances so they knew exactly how much they could ask for, without it bankrupting the business owner (so that the owners company could keep thriving and thus could keep paying protection money).

What? I read the book "Never Split the Difference" written by a former FBI hostage negotiator, and it had stories of him helping with some kidnapping cases and offering a ransom. If I remember correctly his goal was not to avoid paying a ransom, but to make it as small as possible.

Good book by the way.

but this wouldn't be true of governments, compared to private citizens, since government spending is a matter of public record.

That said, this city could have used some negotiation help.

[1] https://www.congress.gov/bill/114th-congress/house-bill/5069...

    the City Council unanimously agreed to have its insurance carrier pay

I'm surprised cybersecurity insurance doesn't mandate best-practice auditable backups as part of the process to grant a policy.

I'm a runner, and recently I've seen a lot of ads for a company called HealthIQ (I think) that offers cheap life insurance, but only for people who can run a 9 minute mile.

I think breaking into health insurance would be much harder because a) the administration is way more complicated and b) most people get health insurance through their employers, and normal employers won't be able to guarantee that every employee can pass a healthiness test, but I imagine they're working on getting around these problems right now.

Fully-integrated HMOs (think Kaiser) have extensive tracking and best practices that reduce future risk and liabilities: well mother / we'll baby care and training, vaccinations and nutrition, preventive chechups, monitoring of dangerous conditions, ob/gyn checkups, breast, colon & prostate exams, etc.

There's only so much that individual initiative can accomplish, but systemic measures really can move the needle.

What government level? The federal government?

> This way it's easier to make sure everything is functioning properly and the cost is only paid once.

The federal government is no paragon of software virtue, nor is it likely to produce software adapted all that well to all of the needs of various states and cities, so what you'd end up with is software less fit for purpose, not particularly free from vulnerability, and where all of the vulnerabilities expose every state and local government in the country rather than just one jurisdiction.

And that's still assuming good intent, but in many cases the state and federal government have adversarial relations on particular issues, which might lead to the federal government actively designing software in a way to frustrate the needs of particular states.

I worked on a system for over a decade that originally got awarded as a contract to build 3 systems. When all was said and done, 3 contracts had been awarded to get it to completion and 1 system (which required 9 months of full time work to get into a state to be used in production) was delivered. There are only 2 words for that in the contracting world: stellar success. That was a bigger win than they could ever have dreamed. Just drag it on, hiring the lowest paid new grads you can find to slap something together, stack the project with absurd layers of management, and collect the checks. Eventually, after years and years of this leeching, someone in the government will decide to make it an achievement in their career that they actually got the thing across the finish line. To do that they will sign off on the project and accept it no matter how short of contract requirements anything is. Their goal is to get the thing in the door and get credit for that, no one is going to blame them when its terrible. And the idea of actually punishing the companies that do this, penalizing them financially and legally for violating their contract when they don't deliver a working system on time? Forget it. Never happen. The companies will get the public fighting against 'big government' and crying crocodile tears for how harangued the billion-dollar megaconglomerates are with the RNC clanging finger cymbals while whirling around chanting 'jobs jobs jobs'.

Think of government as your grandparents. You can give them the best computer and software, but odds are they'll still call you in the middle of the work day to ask questions you don't really have time to answer. This is why Accenture and other big shops get big government contracts. At some point, it's easier to just send Geek Squad to your grandparent's house... knowing full well that they'll get upsold on crap they don't need, and charged more than they should... it's still easier than having to deal with teaching your grandparents not to write their passwords on PostIts they leave next to the computer.

Might be some problems with the competitive bidding aspect of things, as well.

The same is true of private sector. It's not like a bunch of businesses haven't had to pay ransom too you know.

> nobody who knows anything about tech would be caught dead working for local government

People like you are part of the problem.

The problem of what? We seem to be at an equilibrium, non-federal government offers neither the compensation nor the work environment to attract people who have better options. Maybe you can make an argument for civic duty, but that runs into the same problems as working for a company that's "changing the world" but not treating you very well. It just seems like a recipe for burnout, except the tech experience you accumulate probably won't be as good. No?

I think the real issue is the "work environment" aspect you're talking about. The public sector jobs I've worked have not been keen to chase resume-padding fads and have generally much preferred sensible and simple solutions to the complicated over-engineering that is commonly fetishized of our industry.

Anyway, my point about you being the problem is that you're unwilling to put up with whatever you see as the inadequacies of working for local government. You wouldn't be caught dead doing so, in your own words. You care far more about your own personal wealth enrichment than your community, so is it any wonder you end up with a government that isn't any good?

Fair enough, I wasn't paying attention.

> It's far from obvious to me that working for local government means working for "[my] community", especially in my rather scandal-plagued city.

How do you think it is that governments get this way? It's because the people who care about the community don't take part, so it is left to the people who want to exploit it.

Not true at all. Distrust of government is a relatively new phenomenon in American politics. It can be traced back to Reagan's infamous "The most terrifying words in the English language are: I'm from the government and I'm here to help" quote. This is because Reagan strictly believed in small government, and wanted to limit government interference in most things.

Before Reagan, Americans had no issues trusting government to solve big problems or accomplish major goals. See the Space Race, and Roosevelt's New Deal policies two decades before that. American people were largely optimistic about those endeavors because they trusted their government.

If none of these people have the knowledge to make an informed decision, they will defer to either internal IT staff (if that even exists), or their contracted MSP.

I seriously doubt there is much, if any, proactive coordination between the council and IT.

Update: The servicer appears to be Gallagher Basset based on the 2018-19 budget and legal cases cited online.

City records (CC agendas, minutes) are painful if not impossible to navigate.

https://rivierabch.applicantpro.com/jobs/

Ransomware is a type of malicious software designed to block access to a computer system or computer files until a sum of money is paid. Most ransomware variants encrypt the files on the affected computer, making them inaccessible, and demand a ransom payment to restore access.

Ransomware is rarely individually targeted, but rather a “shotgun” approach where the attackers (Clue I) acquire lists of emails or compromised websites and blast out ransomware.

Microsoft used a method to install software giving it superuser rights without a login. (Clue II) Most ransomware is based on this same install job. It is lightweight but identifiable.

Ransomware is a tripartite intruder and is based on what's already there on Windows (mscexe) in your compute and a substitution of legit program (outlook encrypt) Once the 3 parts are there your system is theirs and only a windows product key method "EFHST-G6ERT-VXWMT-FF8MB-MYERR" can free it - all thanks to Microsoft's product key methodology.

Oh and "backups" & PCmatic won't help and because Microsoft uses the same method to stop you from sharing software. You have seen the screen yourself => you have entered an invalid the product key!

Ransomware can be shipped with a NSA crack( EternalBlue ) forced onto the city of Baltimore (Clue III ) but the same code to create a superuser is open to the public is the end to all protection - because it hides using Microsoft's hidden directory method.

Well what to do now, pay the BTC? Yes and NO Yes buy BTC and NO this is where we create a pigeon drop for out NSA connected friends - we don't accept the face price and try to keep our BTC keys and Encrypt theirs.

For the FBI and NSA the profit from robbing Venezuela, Iran, Russia, Ukraine and Switzerland has been too great for them to stop. As witnessed with Venezuelan money gone and power outage.

That said, demand that Microsoft be held liable for product defects and to make all actions visible to the end user community ( no hidden files or directories ).

A city government in Florida is not the US federal government.

It would probably be cheaper to resolve citizen's issues directly for all missing data not present in the backup (even if it's not, that was a crazy decision made by the city council)

That doesn't mean some criminals won't just take the money, but it does mean that most of them wont and that the larger players have a vested interest in keeping that behavior to a minimum.

Every nation gets the government it deserves. - Joseph de Maistre

The common statement is that "democracy is tyranny of the majority*, which is basically true IMO. Tyranny is not synonymous with fascism, though fascism can certainly be a form of tyranny.

Anyway, it doesn't matter if the voters aren't one person; they're a collective, and collectively they generally approve whatever government they're living under, or else they wouldn't have elected it, or allowed it to continue to rule them. If they elected it, they're getting what they voted for and what they deserve. If they didn't elect it, but allow it to rule them anyway, they're still getting what they deserve (though I'd make an exception for a small country being forcibly occupied by a much larger and more powerful country).