Re:BitBucket was down
by kevinburke on Sunday April 27, 2014 @10:25AM
(#7655167)
I doubt they are pushing new configuration at 6am on a Sunday morning.
[ Parent
| on HN ]
I doubted that Github would push new (bad) production code mid-day unannounced, and it still happened. To be fair, Github I think pushes new production code several times a day every day?
Re:BitBucket was down
by kevinburke on Sunday April 27, 2014 @10:44AM
(#7655252)
If you are pushing new configuration at 6am on a Sunday, and things immediately stop working, you revert the configuration change.
If you work in an environment where such rollbacks are that simple, you're in the rare minority. The reality of working with large-scale distributed systems is that rolling back becomes much more complicated. Push out new code and the accompanying DB schema change? Good luck rolling back to the older schema when you find the bug.
I don't think it's that simple, especially if the new bad configuration has already run amok with the machine/data. Backups are a thing, but at that scale, is probably not completely current every minute.