Re:How to lose $172,222 a second for 45 minutes
by candydance on Tuesday October 22, 2013 @01:42AM
(#6589969)
Have them in pairs or teams, war games style.
Guy1 "Sir, we're losing money much faster than predicted, turn your key"
Guy2 "It could just be market variance"
Guy1 pulls out gun "Turn your key, sir"
Re:How to lose $172,222 a second for 45 minutes
by kops on Tuesday October 22, 2013 @05:42AM
(#6590682)
Despite the gun brandishing, that conversation looks too polite to have ever taken place on a trading floor :-)
Re:How to lose $172,222 a second for 45 minutes
by paganel on Tuesday October 22, 2013 @04:35AM
(#6590471)
> running a shadow copy
Honest question, it's that really doable at this scale?
Re:How to lose $172,222 a second for 45 minutes
by babuskov on Tuesday October 22, 2013 @04:56AM
(#6590537)
It's doable. The real question is how much it would cost and does it make sense then.
Re:How to lose $172,222 a second for 45 minutes
by michaelt on Tuesday October 22, 2013 @05:27AM
(#6590640)
In this case the system was deploying new functionality, so your shadow couldn't just run last week's version or you'd get false alarms all the time. And obviously deploying the same broken version to the primary and the shadow wouldn't detect anything.
So you'd need two codebases and two developer teams, coordinated enough that their code produced exactly the same output yet independent enough that they didn't make the same mistakes. With the challenges of coordination this would more than double your costs.
Of course, with the benefit of hindsight, the costs might have been worthwhile...
Since there were no procedures in place, would you like to be the guy who pulled the plug on the (let's guess) $100 million/minute processing system? Do you think you could get another job after that? What would the costs be for violating contracts? You could single handedly sink the company (which, in the end, this issue basically did).
I don't blame the guy for not killing all operations. He never should have been put in that situation. Proper QA, regression testing, monitoring, running a shadow copy and verifying it's output, there are tons of things that could have prevented/mitigated this.