News for hackers. Stuff that matters.
Re:OVH outage explained by ngrilly on Saturday November 11, 2017 @04:11PM (#15678219)
Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down.

I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this.

Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?

Re:OVH outage explained by mioelnir on Sunday November 12, 2017 @06:47AM (#15680549)
Simulated DC failure is more often then not just traffic flow engineering. It is more about testing the DC that takes over the traffic than it is about testing service restart in the inactive DC.

There is little to test about the introduction of a hard fault, but the service resumption in the other DC is full of data to analyze. Also, in such a setup, getting the fault location running again is not on a hard clock, since it is about restoring redundancy instead of the service.

Re:OVH outage explained by cesarb on Saturday November 11, 2017 @08:48PM (#15679240)
> I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure

It was Google: http://queue.acm.org/detail.cfm?id=2371516 ("Weathering the Unexpected" [2012])

Re:OVH outage explained by ngrilly on Monday November 13, 2017 @05:02AM (#15685419)
Thanks! This is the article I had in mind.