Re:Knightmare: A DevOps Cautionary Tale (2014)
by xyzzy123 on Wednesday February 04, 2015 @05:33AM
(#8996144)
With the benefit of hindsight I think company wide realtime dashboard would be high priority. I guess actual deployment procedures would be higher though.
Re:Knightmare: A DevOps Cautionary Tale (2014)
by zhte415 on Wednesday February 04, 2015 @10:11AM
(#8997161)
If the system was not counting its trades, would they show up in the dashboard?
A reconciliation would be necessary, which would come from where the orders were being sent to (the exchange), but with millisecond delays a realtime dashboard seems only necessary for this case (not that it is a bad case) and while End Of Day reconciliations are needed, I'd be interested if anyone knew of exchange requirements for intra-day trading reconciliations?
Re:Knightmare: A DevOps Cautionary Tale (2014)
by geocar on Wednesday February 04, 2015 @08:15AM
(#8996586)
They had a wire-protocol they were incrementally updating without a version-byte.
That is caused by bad engineering, and they needed to do less of that.
"Realtime dashboard" and "deployment procedures" are more engineering, not less.
In hindsight, this would still have been preferable to losing $400 million, but quite obviously nobody at the time realized just how catastrophic this was going be.