A kill switch wouldn't have saved them. What killed Knight wasn't the $400 million loss, it was the lack of confidence all other firms had in them afterwards. Brokers can't just shut down in the middle of the trading day.

They managed to raise the money to cover the loss, but afterwards they were getting round 10% of their normal order volume [1].

Somewhat ironically, the closest thing they had to a kill switch, backing out the code, actually made the situation worse as it made all 8 servers misbehave instead of just the first one[2].

The full SEC report in [2] is an interesting read, just skip the parts about "regulation 5c-15...".

[1] http://www.businessinsider.com/look-how-knight-capitals-trad...

[2] http://www.sec.gov/litigation/admin/2013/34-70694.pdf

Note that the loss of confidence was because a) it went on for 45 minutes, b) the financial loss was large enough that it seriously threatened the ongoing business of the firm. A kill switch would absolutely have helped them as it would have solved both a and b. No, they shouldn't stop fulfilling orders (which, incidentally, happens with order fulfillment. Outages and glitches happen), but the alternative was clearly much, much worse.

And to the backout, they reverted the code on the 7 servers while erroneously leaving the revised configuration, so it wasn't really a kill switch at all. It was frantic fumbling that made things worse.

They would have been better off just shutting the servers down at the first hint of trouble.

An operations group should:

1. know what a normal morning looks like

2. and recognize the abnormality

3. and have the authority to shut down all trading immediately

DevOps is not an excuse to fire your operations staff, it's a requirement that your developers work with and understand your operations staff and vice-versa.

As someone else pointed out, "shut down all trading immediately" isn't so clear cut as "let's stop doing this".

The other aspect is that while I think people should be empowered, shutting down business operations should be decided by more than just an operations team.

Here's an example - one morning on my way to work many years ago, I got a frantic call from our head of operations - "the site is slow", "we're getting DOSed", "I'm going to start blocking netblocks from the major offenders".

I talked him into waiting until I got there, and took a look - sure enough, we're doing 6x our normal traffic, web servers are slowing down, most of it coming from the US. But was it a DoS? Credit card transactions were also up, but not 6x, about 1.5x-2x, so more people were buying, but not proportional to the normal traffic volumes.

A short while later we figured it out - unknownst to us, we had been featured on a major US morning news show, complete with a walkthrough on how to use our site. Millions of people jumped on the site to give it a shot, but many of them abandoned, or the site got too slow for them to purchase on. We fixed it, got it up and running again, and made a ton of money.

But if our operations group had obeyed the "clear" signs of what was going on and just started blocking whole netblocks, we would have lost money and hurt our business.