News for hackers. Stuff that matters.
Re:Netflix was experiencing issues by AYBABTME on Tuesday February 03, 2015 @07:51PM (#8994645)
Really looking forward to read about what happened. I know they have great resiliency measures in place, I'd like to know if they worked, in what measure, how they failed or how they weren't enough. Or if there was anything they could have done.
Re:Netflix was experiencing issues by johansch on Tuesday February 03, 2015 @07:56PM (#8994672)
Great resilience measures? Like depending on AWS that's seemingly going down like one full day per year?

I guess what I'm trying to say is that one the one hand i see netflix having lots out outages, on the other hand being celebrated as some sort of thought leader on resilicency/high availability. Good tech/open source marketing goes a long way in certain crowds, it seems.

It's not like doing lots of stateless transactions/distributing read-only content is a really hard problem by now. Akamai has done it well for ages. Yet Netflix while being celebrated primarily for their tech marketing in this area is struggling.

Re:Netflix was experiencing issues by mikeyouse on Tuesday February 03, 2015 @08:36PM (#8994821)
During the evening hours, Netflix represents over 35% of all internet traffic in the US. That is an insane demand to be able to service, but they've done extremely well in my opinion. Much of their tech is open-sourced leading to much better practices industry-wide and they're transparent when things go wrong.

Here's an example of their preemptive work to deal with the loss of AWS/ELB regions:

http://techblog.netflix.com/2013/12/active-active-for-multi-...

An an example of their post-mortem after deploying Node.js with a bad handler:

http://techblog.netflix.com/2014/11/nodejs-in-flames.html

Is there any other service with a library well over a petabyte in size that delivers multiple terabits/second worldwide to millions of customers with better reliability? Maybe YouTube with the backing of a company worth $300B more than Netflix?

Re:Netflix was experiencing issues by johansch on Tuesday February 03, 2015 @08:37PM (#8994831)
Again you are confusing openness with availability. I do agree they are open. I don't really see the connection between their openness and their performance though.
Re:Netflix was experiencing issues by mikeyouse on Wednesday February 04, 2015 @03:53AM (#8995920)
I think my last paragraph was pertinent to your question, is there anyone providing the volume of traffic to millions of worldwide customers comparable to Netflix who is more reliable?
Re:Netflix was experiencing issues by abalone on Tuesday February 03, 2015 @08:27PM (#8994781)
According to this audit EC2 had just 2.14 total hours downtime last year, the least of any major provider by region.

https://cloudharmony.com/status-1year-group-by-regions

This is HN.. You shouldn't use dodgy language like "seemingly" to avoid doing homework and citing sources.

Re:Netflix was experiencing issues by johansch on Tuesday February 03, 2015 @08:30PM (#8994795)
Then there's S3, and xyz, etc.
Re:Netflix was experiencing issues by abalone on Tuesday February 03, 2015 @08:38PM (#8994832)
S3 had 2.81 hours downtime, according to the report I just linked to. Homework...
Re:Netflix was experiencing issues by thezilch on Tuesday February 03, 2015 @08:39PM (#8994838)
xyz? S3 is on the same link. Even if they are additive -- assume the S3 downtime doesn't overlap with EC2 -- they are still at less than a day, half day, quarter day...
Re:Netflix was experiencing issues by chenshu_ivory on Tuesday February 03, 2015 @08:54PM (#8994892)
What does S3 mean? Seems you guys are working for troubleshooting.
Re:Netflix was experiencing issues by thezilch on Tuesday February 03, 2015 @09:51PM (#8995054)
S3 is an AWS product: http://aws.amazon.com/s3/

johansch is trying to paint AWS -- the culmination of its products (EC2, S3, "xyz", etc) -- as something that is down for day(s) a year. We don't (have to) work for these providers to know the real truth and shut down such FUD.

Re:Netflix was experiencing issues by AYBABTME on Tuesday February 03, 2015 @08:13PM (#8994733)
Great resilience measure, in spite of their choice to use AWS.

Any infrastructure will involve downtime, be it your own hardware or some cloud provider. Probabilities will vary, but they're always there. Despite their hosting solution, Netflix is doing (or showing) extra effort in dealing with the reality of failures. I'm interested in that bit.

Sure it might be a good idea to diversify their cloud providers, although I'm not sure this might be involved here, since AWS seems to be doing fine at this time.

Re:Netflix was experiencing issues by fletchowns on Tuesday February 03, 2015 @08:08PM (#8994711)
Re:Netflix was experiencing issues by pat2man on Tuesday February 03, 2015 @07:59PM (#8994683)
Yes, those types of measures.
Re:Netflix was experiencing issues by ahcox on Tuesday February 03, 2015 @08:47PM (#8994869)
Podcast where Adrian Cockcroft goes into the philosophy and technology behind scaling at Netflix: http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...
Re:Netflix was experiencing issues by dragonshed on Tuesday February 03, 2015 @07:53PM (#8994655)
Worked fine from my iPhone over LTE but not Wii over wifi.

And apparently it started right before my son's designated Sesame Street time. Boo.

Re:Netflix was experiencing issues by rodgerd on Tuesday February 03, 2015 @08:48PM (#8994874)
> And apparently it started right before my son's designated Sesame Street time.

Something to be said for local copies of programs that really matter.

Re:Netflix was experiencing issues by FroshKiller on Wednesday February 04, 2015 @09:54AM (#8997065)
Something something offline viewing?
Re:Netflix was experiencing issues by jijojv on Tuesday February 03, 2015 @07:51PM (#8994647)
AWS status is all green so this is a first. Monkey gone wrong ?
Re:Netflix was experiencing issues by mhuffman on Tuesday February 03, 2015 @08:35PM (#8994814)
Rise of the simian army? I suspected netflix was hiding the "good" AI somewhere (because they haven't been using it on my recommendations).
Re:Netflix was experiencing issues by nattaylor on Tuesday February 03, 2015 @07:42PM (#8994613)
Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity) https://support.netflix.com/en/rest/views/popularbytopcat.js...
Re:Netflix was experiencing issues by nattaylor on Tuesday February 03, 2015 @07:47PM (#8994636)
From their Twitter: https://twitter.com/Netflixhelps/status/562760983790485504

We're aware some members are experiencing technical difficulties on the web site in all regions. We hope to resolve this asap.

Re:Netflix was experiencing issues by jameshk on Tuesday February 03, 2015 @07:43PM (#8994620)
I'm watching House of Cards right now, but it took a few tries for it to start and not throw an error.
Re:Netflix was experiencing issues by thezilch on Tuesday February 03, 2015 @07:56PM (#8994667)
Planet of the Apes? Eager for the postmortem.
Re:Netflix was experiencing issues by wodenokoto on Tuesday February 03, 2015 @08:02PM (#8994694)
i have been getting the issues warning and a limited movie selection on and off for days now.
Re:Netflix was experiencing issues by Aldo_MX on Tuesday February 03, 2015 @07:42PM (#8994616)
Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity)
Re:Netflix was experiencing issues by ericfrederich on Tuesday February 03, 2015 @07:54PM (#8994660)
Chaos monkey strikes?