Am I understanding the last one correctly?

1. Customers complain that they can't install latest version because it's checksum doesn't match what SolarWinds posted

2. The checksum doesn't match because malware has been inserted into the package during build/delivery

3. SolarWinds tells customers to ignore this and install it manually

Did no one think to check why the checksum didn't match?

One suspects they've given this advice for a long time... because their shit has been hacked for a long time.

This seems an unfair leap. The most common cause of a checksum mis-match is going to be a partial download or something similar.

It's also not relevant to the current attack since the code was legitimately included in the official release and, as such, baked into the valid checksum results.

Is the proper response to tell a customer to install the package anyway because it's just a partial download or something similar? Regardless, it seems irresponsible.

#2 is speculation. Seems possible that there's an unrelated bug causing checksum errors. In any event, it's not a good look right now.

Regardless of the motivation, cause, mechanism of #2 - #3 is not the appropriate way to handle the problem. Attack is indistinguishable from unintentional corruption. And #3 trains customers to do the wrong thing when they encounter an attack.

The malicious file was signed with the right certificate. So yeah you should ideally be more careful with checksums but there already was a much more robust and secure authentication mechanism and it was defeated.

Yes, these are two orthogonal egregious security problems.

Solarwinds is def. Used by acrive duty cyber units at Lackland afb...and they wonder why we tell them they can't just install what they feel like.

And you posted this US military vulnerability on a publicly searchable internet site?

head desk

TO be fair, it isn't really secret, if you look at any job posting for lackland, you'll see it mentioned over and over..

https://careers-salientcrgt.icims.com/jobs/11200/network-sys...'

https://i.imgur.com/d8KbSZp.png

But, wow, imagine that's a job, just walk in, look at two programs and swap out parts as needed.

The qualifications reads 'Someone from HR came up with this'

Isn't that true of most postings?

Facebook query Find people who work for US Air Force.

Vulnerabilities publicly available are numerous, and I gave no such details to anyone that would give them an easier time finding said compromises.

Its like saying windows 10 bug found --> HEY THE MILITARY USES WINDOWS 10.

My employer has a knowledgebase on the public internet that is littered with lists of softwares and practices. There are thousands of employees. Name dropping software should be a risky thing to do, but that isn’t the world we live in.

Give the OPSEC snide comment to the job postings publicly advertised.

Don't hate on marines, they do hard work.

Using a throwaway account to be trite seems par for the course tho for opinions that can be disregarded.

> SolarWinds hasn't bothered to revoke their certs or remove the package

Amazing. While I'm sure the attackers have already shut up shop and the threat no longer exists, this feels insanely tone-deaf from SolarWinds.

Maybe they were just bribed?

An employee, possibly. The whole company, unlikely. And either way, even if someone was bribed to introduce the attack there's zero reason to allow the hacked software to be downloaded now.

I work at a large and highly regulated (HIPAA) company and we have the equivalent of Electric Dylan/Pete Seeger with the axe: if someone at the VP+ level declares a major incident, our infosec team has a script that will lock down all inbound/outbound traffic, snapshot all our running machines for later forensics, lock our AWS IAM access down to a single incident response account, and move DNS for our web properties to a "we've been hacked" page. (OK, it obviously doesn't say that, but something similar that has been heavily vetted by legal and marketing ;-)). We've drilled and timed it out and can stop the ship in ~5 minutes.

Either SolarWinds doesn't have a major security incident response plan, or they don't have the stomach to pull the trigger. Neither is promising.

Sounds like a solid information security incident response mechanism!

The only missing piece is making sure that VP+ level folks are not incentivized in any way to suppress incidents. However, that’s beyond infosec—in that treacherous area between information security, shareholder interests and organizational politics.

I wish business continuity planning (which would include infosec procedures but has a much wider overall scope) was paid more attention and more widely scrutinized.

This doesn’t sound like a good incident response plan to me at all, precisely because it provides a very clear incentive to not activate it. If you have to be so sure that you’re having a serious incident that you’re prepared to put a stop to all operations in the organization, then you can be pretty sure that plan is never going to be used.

You’re not going to turn the business off because somebody’s inbox got compromised, or because there’s some unexplained event in the SIEM, and those are the sort of events you’re actually going to have to respond to.

> You’re not going to turn the business off because somebody’s inbox got compromised, or because there’s some unexplained event in the SIEM,

duh, those get handled several pages before "press the red button" is even discussed. You think "turn off the business" is the only page in the playbook?!

> and those are the sort of events you’re actually going to have to respond to.

Tell that to SolarWinds.

You need a IR plan that has appropriate responses to the threats you are facing. But at the scale and impact of a company like SolarWinds it's actually rather reassuring to have a "stop the world" backstop because your threat model absolutely includes catastrophic levels of risk.

And "you won't be incentivized to push the button"? Come on. When things get to "state level adversary on your network, using your software to attack DHS and the Treasury" bad, you're going to absolutely push the button because in a few months when your CEO is answering questions in Congress they'll want to be able to talk about something that went right.

In the real world, you're never going to know that you have a "state level adversary on your network, using your software to attack DHS and the Treasury" until after all the damage has already been done, and you've had enough time to assess the total impact. That's presuming you're even alerted to it in a timely manner. In that scenario, the appropriate response almost certainly not going to be "turn off the business" and even if it is, it's not going to matter whether you can do it in 5 minutes or 5 hours.

The only scenarios in which you'll have enough information to justify activating this plan, are scenarios where you'll also have enough information to respond to the actual threat, rather than just shutting everything down.

It's something that might sound impressive to people who aren't experienced with incident response, but it's practical uses are so close to non-existent, that any time that was spent developing this solution was most certainly wasted in lieu of doing something actually useful.

Considering HIPAA, upper management could see how not invoking this plan, and correspondingly risking more damage by leaving systems open, on balance could be worse than saving pennies and winging it. If the procedures described make it possible to lock everything down fast and gradually resume operations smoothly, the downtime could be short enough.

The situation would have to be so out of hand by that stage that I can’t imagine being able to do it in 5 minutes would matter. For this scenario to make sense, you’d have to know things were really bad, but not know enough about how bad they are to only isolate the systems you need to.

If you don’t know what’s happened, I can’t imagine you’d know enough about the impact to justify turning the business off. The only scenario I can think of where this plan would make sense is if you find out somehow that you’ve already been the victim of a major breach that you failed to detect, so you think it would be worthwhile to just turn everything off while you figure out what happened (because how much worse can it get at that stage, really?...).

Nothing about this seems impressive to me. It sounds like a plan for people who don’t have a plan.

Also, as a side note, anything that needs executive approval to be done during an incident is (as a general rule of thumb) never going to be done during an incident.

>if someone at the VP+ level declares a major incident [...]

I read this as, "we have a policy that under no circumstances will someone at a VP+ level declare a major incident."

Nah. If we ever had to pull this specific trigger we're already in "mandatory disclosure to individuals whose data was breached, the federal government, and possibly the media" territory.

It's one thing to try to duck bad publicity, it's another to not act quickly and risk the ire of the federal government.

1. That's impressive

2. My own knowledge of folk rock and subsequent visits to Google and Wikipedia have not helped me interpret this reference, in this context:

"Electric Dylan/Pete Seeger with the axe"

Help, please :-D

Ha!

https://en.wikipedia.org/wiki/Electric_Dylan_controversy

http://communityvoices.post-gazette.com/arts-entertainment-l...

> The Cliff Notes version is Dylan, whose latest album Bringing It All Back Home had upset many folk purists with its amplified accompaniment, performed at Newport on July 25 with amplified backing by the Paul Butterfield Blues Band, who played the festival on their own. As an offended audience booed Dylan performing with Butterfield's band (minus Butterfield himself), an incensed Seeger, outraged at his friend's apostasy, wanted the audio shut off and sought an axe to cut the cables as Dylan and the band ripped through "Maggie's Farm" and "Like A Rolling Stone," Dylan's just-released single.

TIL! -Bravo!

I guess if you can be as successful as SolarWinds with that level of incompetence I should stop worrying so much about myself.

You'll be surprised at how technically illiterate most corporations are and how marketing and not engineering are responsible for the success for some of the software companies.

You'd be surprised honestly

Clearly whoever is the CIO/CISO could care less? I find it hilarious that people get these positions without seemingly a care in the world. Or maybe they do care and the CEO didn't? Hardly anyone ever gets fired in these circumstances.

Surely he COULDN'T care less?

This is one of those colloquialisms that has become so commonplace that it actually becomes a part of the language rather than being incorrect.

https://www.oxfordlearnersdictionaries.com/us/definition/eng... lists it as a usage in North American English.

Yes, when enough people do / say something wrong over and over again, it sometimes becomes normalised.

Wrong would imply a right, but most interpretations of what is "right" are fairly arbitrary and mostly have to do with social and cultural hierarchy.

Logically it's wrong. If you COULD care less then that means you do care to some level whereas most people mean you COULD NOT care less. Simple...

Those of us who know better bristle and retain pedantry. Consider semi-annual and biannual. Grr.

LOL that last one. Why bother having the checksum at all in this case....

Is it possible that there could be SolarWinds customers who are not vulnerable because, for whatever reason, they did not enable/install updates. Were updates to the Orion software necessary for the original software to continue to function or were they optional.

They've said that 33k customers were potentially exposed but only 18k actually downloaded that update.

https://www.zdnet.com/article/sec-filings-solarwinds-says-18...

They were optional. I know a company still using a version from 2015/2016.