News for hackers. Stuff that matters.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by dusing on Monday October 29, 2012 @12:25PM (#4712981)
"On Monday, just after 8:30am, we noticed that a couple of large websites that are hosted on Amazon were having problems and displaying errors."

Our sys admins couldn't get to reddit.

Re:Netflix: Post-mortem of 22 Oct AWS degradation by jedberg on Monday October 29, 2012 @12:44PM (#4713087)
Actually, that is completely true. I was trying to post a link to my Airbnb talk, noticed reddit was down, and then noticed Airbnb was down too.

One thing we might start doing is actually having alarms when two or more major AWS sites go down.

Re:Netflix: Post-mortem of 22 Oct AWS degradation by samstave on Monday October 29, 2012 @12:50PM (#4713128)
Where is the link to said AirBnB talk?
Re:Netflix: Post-mortem of 22 Oct AWS degradation by jedberg on Monday October 29, 2012 @12:57PM (#4713173)
Re:Netflix: Post-mortem of 22 Oct AWS degradation by azylman on Monday October 29, 2012 @02:30PM (#4713716)
This is completely unrelated to the post, but I was at the Airbnb tech talk and it was extremely interesting - thanks for putting that on!
Re:Netflix: Post-mortem of 22 Oct AWS degradation by snprbob86 on Monday October 29, 2012 @02:00PM (#4713577)
> If you like thinking about high availability and how to build more resilient systems, we have many openings throughout the company

Money can't buy recruiting opportunities like these. This is exemplary engineering and marketing.

Re:Netflix: Post-mortem of 22 Oct AWS degradation by ericcholis on Monday October 29, 2012 @12:56PM (#4713163)
I enjoy reading these Sysops articles from Netflix. They provide a pretty good blueprint for working inside the cloud.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by 3amOpsGuy on Monday October 29, 2012 @02:07PM (#4713613)
They are good. I'd enjoy reading more from their ops guys - likely others wouldn't though :-( a significant portion of ops concerns would probably appear unsexy to non ops people, but horses for courses, whatever floats your boat and all that. I love it.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by waven on Monday October 29, 2012 @04:34PM (#4714324)
3amOpsGuy, is there any chance I could email/contact you somehow regarding some ops advice? thanks!
Re:Netflix: Post-mortem of 22 Oct AWS degradation by 3amOpsGuy on Monday October 29, 2012 @05:37PM (#4714608)
Re:Netflix: Post-mortem of 22 Oct AWS degradation by confluence on Monday October 29, 2012 @08:59PM (#4715479)
I saw a comment a few weeks ago where one fellow HNer ran his entire startup on AWS spot instance pricing - so that he was forced to program in a state of continuous chaos as his demand/spot instances popped randomly and continuously into and out of existence while his service was running. It's like programming on quicksand.

This is probably a step too far - but maybe it is a natural extension of NFLX's Chaos monkey.

If you want ~100% up time with no QOS degradation - your system must be constantly under catastrophic attack.

This is probably a major reason why vol based risk models in finance are completely pointless.

Value at risk of any investable securities (including cash) is 100% all the time - any other number is bullshit. All volatility based risk models are useful if you like watching squiggly lines or pricing options, but they are essentially a random anchor that helps us sleep at night (anchoring bias).

Re:Netflix: Post-mortem of 22 Oct AWS degradation by signifiers on Monday October 29, 2012 @03:32PM (#4714029)
Key line of the article: “Since Netflix focuses on making sure services can handle individual instance failure and since we avoid using EBS for data persistence, we still did not see any impact to our service.”

As an architecture design, the choice to avoid EBS is hotly debated, though many high-profile systems besides NetFlix (SimpleGeo, Sprint.ly) have moved almost exclusively to EC2 instance-backed (local disk) VMs and as a result, avoided the pain of the last 3 major AWS outages.

Re:Netflix: Post-mortem of 22 Oct AWS degradation by EzGraphs on Monday October 29, 2012 @03:08PM (#4713910)
Netflix has spent a lot of time and energy devising solutions that minimize disruptions. Are any of you Cloud-Savvy folks using Netflix's tools on Amazon? If so, are you using them off-the-shelf or did you need to customize them to suit your site? In particular:

Asgard is cited in the post as making a "zone evacuation" relatively straightforward.

Astyanax (their Cassandra client) is designed with "smarts" that allow it to choose from available nodes should one or more be unavailable.

These (and several other tools) are available from Netflix at Github:

https://github.com/Netflix

Re:Netflix: Post-mortem of 22 Oct AWS degradation by shuw on Monday October 29, 2012 @01:02PM (#4713214)
If this practice was widely adopted, I wonder if AWS would experience a bank run/DOS in the affected and neighboring zones.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by jedberg on Monday October 29, 2012 @01:16PM (#4713296)
It would certainly make reservations a lot more important.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by Terretta on Tuesday October 30, 2012 @07:47AM (#4716986)
You don't have to wonder. Every single AZ AWS postmortem says that's exactly what happens.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by smoyer on Monday October 29, 2012 @06:47PM (#4714952)
My impression is that the Netflix team understands AWS better than Amazon does ... But certainly better than most other AWS customers. Kudos
Re:Netflix: Post-mortem of 22 Oct AWS degradation by ams6110 on Monday October 29, 2012 @08:33PM (#4715377)
OTOH, most AWS customers are not Netflix and couldn't afford the sort of high availability architecture Netflix has.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by smoyer on Tuesday October 30, 2012 @08:07AM (#4717025)
Not "OTOH" ... what you've said is the truth. Some of the methods Netflix uses still apply but others are indeed financially non-viable. I'd love to have a Simian Army of my own and I think that would translate to any PAAS provider.
Re:Netflix: Post-mortem of 22 Oct AWS degradation by crb on Monday October 29, 2012 @02:36PM (#4713749)
One of the more interesting findings from the most recent AWS outages is that Elastic Load Balancing (ELB), the best-practice way to handle multi-AZ deployment, has a dependency on EBS. I know from various talks that Netflix try not to use EBS these days, but I wonder if you had any ELB problems, and how you might have coped if you had evacuated a zone but your LBs were out?