Re:Finance world in the dark as Bloomberg terminals go offline
by tomp on Friday April 17, 2015 @04:04PM
(#9396871)
That's very interesting, and definitely an infrastructure many companies should strive for and could learn from.
I'm wondering, however, not how problems were handled, but how were they detected - code throwing exceptions is the easy case, the real problem is code running almost correctly, but with slightly wrong outputs. I imagine there could be many different "sensibility" checks implemented, (e.g. no negative prices, no huge jumps price jumps), but in general it appears to be a very interesting and complex problem.
What I remember from my time there (2003-2007) was how quickly everyone was able to ship code despite (or perhaps because of) no global style guide, unit test mandates, or even rules about copy-pasting code. Each team was able to do things the way they thought best, thanks to an incredibly resilient, flexible production infrastructure that could tolerate, isolate, route around, and quickly repair any newly-introduced bug.
The code path (Bloomberg four-letter function, to those familiar with the service) running through the bug could nearly instantly, and if I'm not mistaken, in a totally automated way, be flipped over to a hot backup instance running the previous week's code, and a patch or rollback could be released to the live code almost as soon as it was written.
Most tech companies see the problems caused by coding too fast and respond by doing things to slow people down. Bloomberg cured the ills of excess programming velocity by turning the speed up even further. Like Facebook's "Move fast and break things", but with an extra, "...and then move even faster to fix them", and in a way that (mostly) insulated customers from being exposed to the breakage.
The fact that it's front-page news when there's a hiccup shows that it's a man-bites-dog event. You'd be in trouble if it wasn't major news when there was a service incident.