An employee, possibly. The whole company, unlikely. And either way, even if someone was bribed to introduce the attack there's zero reason to allow the hacked software to be downloaded now.

I work at a large and highly regulated (HIPAA) company and we have the equivalent of Electric Dylan/Pete Seeger with the axe: if someone at the VP+ level declares a major incident, our infosec team has a script that will lock down all inbound/outbound traffic, snapshot all our running machines for later forensics, lock our AWS IAM access down to a single incident response account, and move DNS for our web properties to a "we've been hacked" page. (OK, it obviously doesn't say that, but something similar that has been heavily vetted by legal and marketing ;-)). We've drilled and timed it out and can stop the ship in ~5 minutes.

Either SolarWinds doesn't have a major security incident response plan, or they don't have the stomach to pull the trigger. Neither is promising.

Sounds like a solid information security incident response mechanism!

The only missing piece is making sure that VP+ level folks are not incentivized in any way to suppress incidents. However, that’s beyond infosec—in that treacherous area between information security, shareholder interests and organizational politics.

I wish business continuity planning (which would include infosec procedures but has a much wider overall scope) was paid more attention and more widely scrutinized.

This doesn’t sound like a good incident response plan to me at all, precisely because it provides a very clear incentive to not activate it. If you have to be so sure that you’re having a serious incident that you’re prepared to put a stop to all operations in the organization, then you can be pretty sure that plan is never going to be used.

You’re not going to turn the business off because somebody’s inbox got compromised, or because there’s some unexplained event in the SIEM, and those are the sort of events you’re actually going to have to respond to.

> You’re not going to turn the business off because somebody’s inbox got compromised, or because there’s some unexplained event in the SIEM,

duh, those get handled several pages before "press the red button" is even discussed. You think "turn off the business" is the only page in the playbook?!

> and those are the sort of events you’re actually going to have to respond to.

Tell that to SolarWinds.

You need a IR plan that has appropriate responses to the threats you are facing. But at the scale and impact of a company like SolarWinds it's actually rather reassuring to have a "stop the world" backstop because your threat model absolutely includes catastrophic levels of risk.

And "you won't be incentivized to push the button"? Come on. When things get to "state level adversary on your network, using your software to attack DHS and the Treasury" bad, you're going to absolutely push the button because in a few months when your CEO is answering questions in Congress they'll want to be able to talk about something that went right.

In the real world, you're never going to know that you have a "state level adversary on your network, using your software to attack DHS and the Treasury" until after all the damage has already been done, and you've had enough time to assess the total impact. That's presuming you're even alerted to it in a timely manner. In that scenario, the appropriate response almost certainly not going to be "turn off the business" and even if it is, it's not going to matter whether you can do it in 5 minutes or 5 hours.

The only scenarios in which you'll have enough information to justify activating this plan, are scenarios where you'll also have enough information to respond to the actual threat, rather than just shutting everything down.

It's something that might sound impressive to people who aren't experienced with incident response, but it's practical uses are so close to non-existent, that any time that was spent developing this solution was most certainly wasted in lieu of doing something actually useful.

Considering HIPAA, upper management could see how not invoking this plan, and correspondingly risking more damage by leaving systems open, on balance could be worse than saving pennies and winging it. If the procedures described make it possible to lock everything down fast and gradually resume operations smoothly, the downtime could be short enough.

The situation would have to be so out of hand by that stage that I can’t imagine being able to do it in 5 minutes would matter. For this scenario to make sense, you’d have to know things were really bad, but not know enough about how bad they are to only isolate the systems you need to.

If you don’t know what’s happened, I can’t imagine you’d know enough about the impact to justify turning the business off. The only scenario I can think of where this plan would make sense is if you find out somehow that you’ve already been the victim of a major breach that you failed to detect, so you think it would be worthwhile to just turn everything off while you figure out what happened (because how much worse can it get at that stage, really?...).

Nothing about this seems impressive to me. It sounds like a plan for people who don’t have a plan.

Also, as a side note, anything that needs executive approval to be done during an incident is (as a general rule of thumb) never going to be done during an incident.

>if someone at the VP+ level declares a major incident [...]

I read this as, "we have a policy that under no circumstances will someone at a VP+ level declare a major incident."

Nah. If we ever had to pull this specific trigger we're already in "mandatory disclosure to individuals whose data was breached, the federal government, and possibly the media" territory.

It's one thing to try to duck bad publicity, it's another to not act quickly and risk the ire of the federal government.

1. That's impressive

2. My own knowledge of folk rock and subsequent visits to Google and Wikipedia have not helped me interpret this reference, in this context:

"Electric Dylan/Pete Seeger with the axe"

Help, please :-D

Ha!

https://en.wikipedia.org/wiki/Electric_Dylan_controversy

http://communityvoices.post-gazette.com/arts-entertainment-l...

> The Cliff Notes version is Dylan, whose latest album Bringing It All Back Home had upset many folk purists with its amplified accompaniment, performed at Newport on July 25 with amplified backing by the Paul Butterfield Blues Band, who played the festival on their own. As an offended audience booed Dylan performing with Butterfield's band (minus Butterfield himself), an incensed Seeger, outraged at his friend's apostasy, wanted the audio shut off and sought an axe to cut the cables as Dylan and the band ripped through "Maggie's Farm" and "Like A Rolling Stone," Dylan's just-released single.

TIL! -Bravo!