News for hackers. Stuff that matters.
Re:OVH outage explained by ngrilly on Saturday November 11, 2017 @08:04AM (#15676171)
I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey.

Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network core is not really redundant either.

I'd be curious to know how competitors like AWS and Google Cloud mitigate this kind of problems.

Re:OVH outage explained by vbernat on Saturday November 11, 2017 @10:04AM (#15676602)
Restarting a whole datacenter from scratch is different from restarting individual servers. If a service needs another service running on another server, it may fail to start. For example, many servers may need the DHCP server. Some servers may need the DNS. Some others may need Kafka to be up and running, etc. You may have some documentation on which servers to start first, but things are moving so fast that it's not unlikely you'll need to rely on supervision to know which service was able to start correctly.
Re:OVH outage explained by ngrilly on Saturday November 11, 2017 @01:44PM (#15677505)
I don't buy this argument. If a service needs another service running on another server which is not already available, it should retry with an exponential backoff. Doing anything else is a bug.
Re:OVH outage explained by snom380 on Saturday November 11, 2017 @03:53PM (#15678131)
I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or maybe during the outage power was cycled on some equipment one too many times, causing some failsafe to trip, or maybe some of the core infrastructure like your switches and routers expect some core service to be up, but that core service can't get online without working network.
Re:OVH outage explained by ngrilly on Saturday November 11, 2017 @04:11PM (#15678219)
Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down.

I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this.

Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?

Re:OVH outage explained by mioelnir on Sunday November 12, 2017 @06:47AM (#15680549)
Simulated DC failure is more often then not just traffic flow engineering. It is more about testing the DC that takes over the traffic than it is about testing service restart in the inactive DC.

There is little to test about the introduction of a hard fault, but the service resumption in the other DC is full of data to analyze. Also, in such a setup, getting the fault location running again is not on a hard clock, since it is about restoring redundancy instead of the service.

Re:OVH outage explained by cesarb on Saturday November 11, 2017 @08:48PM (#15679240)
> I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure

It was Google: http://queue.acm.org/detail.cfm?id=2371516 ("Weathering the Unexpected" [2012])

Re:OVH outage explained by ngrilly on Monday November 13, 2017 @05:02AM (#15685419)
Thanks! This is the article I had in mind.
Re:OVH outage explained by icebraining on Saturday November 11, 2017 @08:08AM (#15676185)
OVH runs servers for their clients, do you really expect them to randomly restart their clients' servers? Netflix is not a hosting provider, the situations are not comparable.
Re:OVH outage explained by ngrilly on Saturday November 11, 2017 @08:14AM (#15676210)
The comment in the OVH post-mortem suggests it is about internal servers and services. You're right about customer bare-metal servers. They can't restart them randomly.
Re:OVH outage explained by wereHamster on Saturday November 11, 2017 @08:15AM (#15676215)
I would be pissed if my hosting provider randomly restarted my servers, taking them offline for minutes each time.
Re:OVH outage explained by drdebug on Saturday November 11, 2017 @08:54AM (#15676360)
So would I, but that actually makes me realize that I should be doing more of what netflix is doing too and ensure I can randomly restart servers without affecting service.
Re:OVH outage explained by tete on Sunday November 12, 2017 @07:35AM (#15680652)
Isn't choasmonkey just randomly restarting virtual servers on Amazon?

Kudos to Netflix, but restarting a virtual server vs a physical or a whole data center are different things.

I think every company that cares a bit about high availability knocks out stuff randomly or at least in different ways, does stuff like introducing packet loss, etc. It's another layer and another thing to test that on service/virtual server layer than on close to physical layers.

Of course, one should test that too and it's nowhere near impossible, but Chaosmonkey is for a somewhat different use case.

Also the "article" mentions that tests are done.

Re:OVH outage explained by ngrilly on Saturday November 11, 2017 @01:23PM (#15677411)
Some of you are commenting that asking OVH to randomly restart customer servers would be silly. I agree.

My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming back gradually since 10:30 am. Our monitoring system allows us to know the list of servers that have successfully started up and those that still have a problem. We intervene on each of these servers to identify and solve the problem that prevents it from restarting."

The language used suggests that these are servers monitored and operated by OVH, not by random customers.

Re:OVH outage explained by Feld0 on Monday November 13, 2017 @02:55AM (#15684965)
OVH monitors customers' servers and investigates if they stop responding to pings, unless you opt out of this. I'm guessing they were looking at this system to make sure all their customers' machines came online again after the incident.

Source: am an OVH customer

Re:OVH outage explained by ngrilly on Monday November 13, 2017 @03:08AM (#15685010)
Yes, I'm aware of this monitoring service. Is OVH able to login in your server to investigate what is the issue?
Re:OVH outage explained by pmontra on Saturday November 11, 2017 @09:47AM (#15676554)
I have some VPSes there. All of them run independent services, no redundancy.

They can restart all the servers they want if they manage not to kill any of my processes, break network connections, change IP addresses. If they can't (it's probably hard) it's up to me to implement a reduntant architecture. Then maybe I'll also implement some random kill switch to proof that the system is resilient to failures.

Re:OVH outage explained by dboreham on Sunday November 12, 2017 @12:06AM (#15679748)
Meanwhile in the real world...