I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this.
Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?
There is little to test about the introduction of a hard fault, but the service resumption in the other DC is full of data to analyze. Also, in such a setup, getting the fault location running again is not on a hard clock, since it is about restoring redundancy instead of the service.
It was Google: http://queue.acm.org/detail.cfm?id=2371516 ("Weathering the Unexpected" [2012])
Kudos to Netflix, but restarting a virtual server vs a physical or a whole data center are different things.
I think every company that cares a bit about high availability knocks out stuff randomly or at least in different ways, does stuff like introducing packet loss, etc. It's another layer and another thing to test that on service/virtual server layer than on close to physical layers.
Of course, one should test that too and it's nowhere near impossible, but Chaosmonkey is for a somewhat different use case.
Also the "article" mentions that tests are done.
My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming back gradually since 10:30 am. Our monitoring system allows us to know the list of servers that have successfully started up and those that still have a problem. We intervene on each of these servers to identify and solve the problem that prevents it from restarting."
The language used suggests that these are servers monitored and operated by OVH, not by random customers.
Source: am an OVH customer
They can restart all the servers they want if they manage not to kill any of my processes, break network connections, change IP addresses. If they can't (it's probably hard) it's up to me to implement a reduntant architecture. Then maybe I'll also implement some random kill switch to proof that the system is resilient to failures.
Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network core is not really redundant either.
I'd be curious to know how competitors like AWS and Google Cloud mitigate this kind of problems.