Re:Lessons Netflix Learned from the AWS Storm
by dangrossman on Friday July 06, 2012 @03:34PM
(#4209286)
You're doing exactly what that article was describing, and calling Netflix's engineering team irresponsible children is a cheap insult. It's extremely insulting that in the process of saying you didn't insult them, you call them children again. So is repeatedly setting up and taking jabs at this straw man that they had no failover system at all, a year ago or last week.
You're basically saying "if I were in charge, Netflix would've been able to fail over to another region in 6 seconds". But you don't even have a fraction of the background info required to say such a thing. It doesn't matter that you've done reliability at Quantcast; Quantcast is not Netflix.
Re:Lessons Netflix Learned from the AWS Storm
by brown9-2 on Friday July 06, 2012 @04:09PM
(#4209400)
Perhaps Netflix merely made a judgment call about the amount of effort required to be able to migrate 100% of their production traffic from one region to another in event of downtime.
It sounds like they came to a different decision than Quantcast about the importance of being able to do so.
Which is more likely - that the thought of an AWS outage like this never occurred to them, or that they judged the specific remedy needed to overcome it not a high enough of a priority to have it available within the first years of their AWS usage?
Curious, how did you come to possess enough knowledge of the Netflix application architecture to believe the techniques that have served you well at Quantcast could unconditionally apply to Netflix?
Re:Lessons Netflix Learned from the AWS Storm
by tedunangst on Friday July 06, 2012 @09:59PM
(#4210453)
Obviously, it works at quantcast. How could it not work elsewhere?
Re:Lessons Netflix Learned from the AWS Storm
by smartwater on Friday July 06, 2012 @09:45PM
(#4210421)
He would make the necessary architecture changes. It wouldn't have to be unconditional.
Re:Lessons Netflix Learned from the AWS Storm
by snowwrestler on Friday July 06, 2012 @04:30PM
(#4209507)
I think it's worth considering that Netflix is the single largest source of downstream Internet traffic in all of North America. The most recent estimates I could find--from about a year ago--are that at peak, Netflix streaming consumes between about 20% and 30% of all NA available downstream bandwidth. So while Quantcast is obviously a highly available service, it seems likely to me the server resources involved are probably quite a bit higher at Netflix.
Re:Lessons Netflix Learned from the AWS Storm
by dangrossman on Friday July 06, 2012 @04:43PM
(#4209563)
The video stream itself is coming off a separate CDN, not their EC2 infrastructure, AFAIK. So that's probably not part of the comparable stuff. What's running on EC2 is the website, the authentication services for all the devices they support, the API for websites and all those devices they support, the recommendation engine, and perhaps a couple million people pinging them every few seconds with the status of their streams. I don't know how that works, but they must be talking to something since reloading a stream on any device will resume where you left off.
Even a child knows that when the power fails, you can't turn on the TV. This isn't specialized technical knowledge.
EDIT: I'm not going to keep responding to your comment below. I'm certain that if I were involved in the design of Netflix' infrastructure, they would be able to survive problems that affect whole regions. (AND I DONT SEE THAT NETFLIX NEEDS 6 SECOND FAILOVER, THERE ARE MUCH EASIER WAYS TO DO IT IN MINUTES).
EDIT: My repeated, emphatic comments are intended to serve a purpose. Everyone should be aware that this is a real problem and you need to plan for it, and it's pretty clear from the discussions on HN that people are surprised by the Amazon downtime. I personally think Amazon does a fantastic job as it is, and Amazon's reliability issues are not an excuse for the downtime of their customers.