Re:Ask HN: Best setups to avoid availability outages on AWS
by rbranson on Saturday June 30, 2012 @09:34AM
(#4181572)
This is correct; probability works against you if you spread your instances over multiple zones without also compensating with some very solid failover. If you are not willing to put the effort into truly going fully active/active/active across THREE availability zones, you might as well just stick with one and weather the outages.
Re:Ask HN: Best setups to avoid availability outages on AWS
by mechanical_fish on Saturday June 30, 2012 @12:01PM
(#4181945)
I think that's a bit of an exaggeration. My old company (Acquia) kept a lot of customers up through the various outages with mere two-zone redundancy.
You do need to actually engineer for failover and recovery, though. That point is correct: It doesn't necessarily help you to have half of your servers survive an outage if that still kills your service or requires painful amounts of manual intervention. And it really is a lot easier to just shrug, keep your backups up to date, and pray for your zone to come back ASAP. Do your cost-benefit analysis.
Re:Ask HN: Best setups to avoid availability outages on AWS
by bfisher9 on Saturday June 30, 2012 @09:17AM
(#4181541)
what if Amazon DNS goes down? Doesn't matter how many zones you've got... DNS remains a potential single (yes it is redundant with HA), but it's still a single point of failure.
I'm in one of the us-east zones and I haven't had a failure in at least a year. They retired one machine I was using and dealing with that was as simple as starting and stopping -- at a time I chose.
With five zones in U.S. East, the probability of a zone failure affecting a single zone systems is 1 in 5.
If you're a busybody who spreads your system across five zones, the probability of a failure affecting you becomes 1.
You're spending more money, and dealing with a lot more complexity, all to increase the probability that hardware failures will affect you.
Now, you're hoping that a zone-distributed system will be able to recover from failures, but that's tricky to do and it's quite unlikely that this will work if you haven't tested it. Add the fact that all the other "cool kids" will be trying to recover their systems at this time and make AMZN's control plane go down.
In the meantime, with probability 4/5 I'm sleeping through the disaster and the first time I hear about it is on hacker news.