News for hackers. Stuff that matters.
Re:Amazon was Down by podperson on Monday April 22, 2013 @01:49AM (#5588007)
Somewhat off-topic: my (limited) experience with Amazon Prime video suggests it's significantly less reliable than Netflix or iTunes (neither of which are stupendously reliable, but I'd say Netflix is by far the most reliable of the three). Hulu might actually be worse than Amazon Prime.
Re:Amazon was Down by notimetorelax on Monday April 22, 2013 @02:03AM (#5588048)
I don't know if you saw this posted on HN, but Netflix test their system really well. They use so-called chaos monkey [1] that shuts down random servers on a whim. This allows them to detect and get rid of dependencies, i.e. tolerate failures in other parts of the system.

[1] http://techblog.netflix.com/2011/07/netflix-simian-army.html

Re:Amazon was Down by jdrobins2000 on Monday April 22, 2013 @04:01AM (#5588338)
Wish I had thought of chaos monkey, much cooler than whatever I called my version of it.

A few years ago I built an automated test system in perl, complete with message bus and message listener container for running tasks on various servers. One of the automated tests I wrote had a component that would periodically (at random intervals) kill processes, unmount shared filesystems, offline interfaces, etc. to cause failovers, to verify that all processes and resources were failed over, and all tasks were reassigned to other nodes and no jobs were dropped or stalled.

It is really the only way to ensure you've covered your bases - beating the shit out of your system repetitively. It uncovered a bunch of big holes and some very obscure ones too, and once we got those fixed it ran pretty much flawlessly.

Re:Amazon was Down by smackfu on Monday April 22, 2013 @09:00AM (#5589054)
The interesting problem is that the underlying AWS system seems to come up with more and more interesting failure modes due to system complexity, that the testing could never catch. Like Netflix had a major outage recently on Christmas Eve 2012.

http://techblog.netflix.com/2012/12/a-closer-look-at-christm...

Re:Amazon was Down by mistermumble on Monday April 22, 2013 @02:06AM (#5588057)
Isn't Netflix hosted on AWS?
Re:Amazon was Down by svedlin on Monday April 22, 2013 @02:25AM (#5588098)
Netflix is indeed running on AWS. Some details about their back-end here:

http://techblog.netflix.com/2012/12/aws-reinvent-was-awesome...

Re:Amazon was Down by jherrick on Monday April 22, 2013 @08:52AM (#5589034)
They also constant kill machines (and replace them with fresh instances of the image) that participate in key load-balanced activities.
Re:Amazon was Down by incision on Monday April 22, 2013 @07:23AM (#5588783)
My experience is the opposite and I use both services regularly. At least one fairly recent event corroborates this [1].

1: http://gigaom.com/2012/12/25/christmas-eve-aws-outage-stings...