News for hackers. Stuff that matters.
Re:AWS US East is experiencing high error rates on several s... by nasalgoat on Sunday September 20, 2015 @09:01AM (#10247353)
This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in.

At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.

Re:AWS US East is experiencing high error rates on several s... by cle on Sunday September 20, 2015 @12:04PM (#10247991)
How would this have helped? In the event of an AWS outage, you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. This outage lasted a few hours, wouldn't it take you that long to switch providers anyway? If you don't abdicate responsibility for maintaining the infrastructure, you lose a lot of the operational benefits of using a cloud service...in general, I think it makes a lot more sense to embrace whatever cloud provider you're using (provided you made a good choice) so that you can focus your efforts on your product and not on maintaining infrastructure.
Re:AWS US East is experiencing high error rates on several s... by kordless on Sunday September 20, 2015 @12:33PM (#10248091)
> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure

No, they are fixing their infrastructure. The point here is that all single provider systems are destined for periodic failure. Not relying on one single provider is, in theory, a service's means of providing higher reliability. This is the general argument for regions and availability zones on Amazon, but that relies on trusting there is no single point of failure with the system (i.e. inside Amazon).

I've worked for several companies who run the majority of their services on AWS, yet maintain functional systems on other providers in the case of an AWS outage.

Re:AWS US East is experiencing high error rates on several s... by rdtsc on Sunday September 20, 2015 @05:27PM (#10249206)
> wouldn't it take you that long to switch providers anyway?

It might, or might not, depends how you built it.

> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure.

And it still was out for hours... You might see it as a great success, but you can also interpret it as "what good is it, even if with so many people behind the scenes they still failed for hours...".

The problem is that because that services ends up being embedded in so many services and products, half the internet ends being down. Even Amazon dogfoods their own stuff so services rely on each other. Dynamo is down, maybe SQS will be down or analytics as well.

Re:AWS US East is experiencing high error rates on several s... by dialtone on Sunday September 20, 2015 @10:47AM (#10247698)
This really is the first major failure of DynamoDB since when I started to use it, when it was in beta over 4 years ago. Not sure many companies would be able to run a multi-million QPS scale database with such a great availability record. In any case we just moved traffic away from us-east-1 and into us-west-2 and all is fine still so you should design at least your critical systems for regional failures.
Re:AWS US East is experiencing high error rates on several s... by jbeda on Sunday September 20, 2015 @10:19AM (#10247588)
Very good point. I wrote a blog post on this topic to try and characterize different types of lock-in. http://www.eightypercent.net/post/types-of-lock-in.html

This was as also part of the strategy when we decided to open source kubernetes. Having an open alternative made the commercial offering (Google Container Engine -- GKE) much stronger because of the reduced dev lock-in.

Re:AWS US East is experiencing high error rates on several s... by acdha on Sunday September 20, 2015 @09:25AM (#10247417)
I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having to maintain more of your infrastructure directly.
Re:AWS US East is experiencing high error rates on several s... by gshx on Sunday September 20, 2015 @11:41AM (#10247907)
Running multi-region services has associated costs. It's also non-trivial engineering effort to convert most services to a multi-region setup. It's definitely worth the investment though. Another option is to run multi-region on the same provider (like AWS) without using proprietary services like DynamoDb but maybe something similar like Cassandra.
Re:AWS US East is experiencing high error rates on several s... by zurn on Sunday September 20, 2015 @11:03AM (#10247753)
There are many high profile companies down (airbnb, IMDb, tinder, ...) so apparently this is not so straightforward.
Re:AWS US East is experiencing high error rates on several s... by peterjancelis on Sunday September 20, 2015 @11:14AM (#10247807)
I don't see airbnb or IMDB being down. If there's some advice on what small time apps (like mine) can do to get thing up again sooner, please let me know.
Re:AWS US East is experiencing high error rates on several s... by cguess on Sunday September 20, 2015 @11:41AM (#10247900)
Explains the weird 500's i was getting when trying to do a review this morning.
Re:AWS US East is experiencing high error rates on several s... by speleding on Sunday September 20, 2015 @10:11AM (#10247560)
In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)
Re:AWS US East is experiencing high error rates on several s... by sitharus on Sunday September 20, 2015 @02:27PM (#10248466)
One of the first things Amazon tell you about AWS is the API is king - the console is hosted in us-east-1 but the API is hosted in each region independently.

You don't need automatic failover, but you do need your own scripts to talk to the AWS API. The whole point of AWS is automation.

Re:AWS US East is experiencing high error rates on several s... by acdha on Sunday September 20, 2015 @10:31AM (#10247630)
AWS sign-in appears to be working but in general it's true that you need some sort of automatic failover. However, isn't that true for every option other than accepting downtime? It doesn't seem like an AWS-specific challenge other except at the implementation level.
Re:AWS US East is experiencing high error rates on several s... by mark_l_watson on Sunday September 20, 2015 @09:50AM (#10247491)
I don't much use it anymore but this is something I always liked about AppEngine: baked into the pricing is automatic replication over three regions.

Serious AWS web apps are also distributed across multiple regions/availability zones, but many companies don't do that. It would be good for hosting services like Heroku, which I use, would replicate across zones also.

To be fare to Amazon, they tell you you architect and built across availability zones.

Re:AWS US East is experiencing high error rates on several s... by hox on Sunday September 20, 2015 @10:09AM (#10247552)
dynamo db is affected in the entire region, however, not just a single AZ.
Re:AWS US East is experiencing high error rates on several s... by DinkyG on Sunday September 20, 2015 @02:51PM (#10248551)
That's why parent post mentioned multiple regions. No other AWS region was affected...
Re:AWS US East is experiencing high error rates on several s... by alexbilbie on Sunday September 20, 2015 @09:12AM (#10247376)
DynamoDB now support cross-region replication [0] so you can build more resilient applications with it

[0] http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

Re:AWS US East is experiencing high error rates on several s... by andrewchilds on Sunday September 20, 2015 @09:25AM (#10247414)
I don't think cross-region replication would've helped in this case:

"The replica tables are intended to serve as read-only copies of the data; however, it is possible to write data to a replica table. If you write data to a replica, those changes will not be propagated to the master, or to any other replicas."

Re:AWS US East is experiencing high error rates on several s... by dialtone on Sunday September 20, 2015 @02:16PM (#10248428)
You can relatively trivially build multi-master cross-region replication in DynamoDB by using kinesis and writing to kinesis instead of DynamoDB directly. On the consuming end of Kinesis you then fan out to all the DynamoDB (or whatever other database you want to use) regions[1]. Admittedly this only works with some relatively relaxed constraints on the latency you can see, intra region latency can go up to 1 second although rarely, while cross-region is around 3 seconds. An important role is also played by the structure of your objects and how accepting they are of concurrent updates coming from different regions (which is the main reason why the default replication in DynamoDB is not multi-master).

[1]: http://tech.adroll.com/blog/data/2015/06/26/kinesis.html

Re:AWS US East is experiencing high error rates on several s... by abalone on Sunday September 20, 2015 @10:35AM (#10247644)
You could at least fall back to a read-only mode. That could be very helpful, compared to going down completely.
Re:AWS US East is experiencing high error rates on several s... by DinkyG on Sunday September 20, 2015 @02:40PM (#10248512)
DynamoDb was only down in only one region, for the first time in years. It's hardly a reason to migrate off, especially with DynamoDb still being available in every single other region throughout. It's simpler to fail over to another AWS region than to an entirely different cloud provider with a different API.
Re:AWS US East is experiencing high error rates on several s... by njharman on Sunday September 20, 2015 @09:20AM (#10247398)
Source for service with better uptime than Amazon?
Re:AWS US East is experiencing high error rates on several s... by mattkrea on Sunday September 20, 2015 @09:51AM (#10247495)
I believe he meant things like using RabbitMQ instead of SQS (SQS isn't very good anyway IMHO) and things like that. More services running on plain old instances rather than AWS specific APIs, Azure specific APIs, etc.

This prevents you from getting too locked in and losing control over your application.

I honestly can't believe that Netflix can't even load their home page without DynamoDB and all this other stuff.

Edit: Looks like Netflix is back online (10:53AM EST)

Re:AWS US East is experiencing high error rates on several s... by nasalgoat on Sunday September 20, 2015 @10:27AM (#10247613)
Yes, exactly. It's tempting to drink the Kool-Aid on the various services but the more you rely on them, the less flexible you become and more open to overall service failures like this one. You can take advantage of the benefits of EC2 without relying on the other services.

I mean, if even Netflix can't stay up during this, what hope does a startup have?

Re:AWS US East is experiencing high error rates on several s... by ukd1 on Sunday September 20, 2015 @10:33AM (#10247639)
Netflix -IS- up, and working fine. Pretty amazing, imho.
Re:AWS US East is experiencing high error rates on several s... by mattkrea on Sunday September 20, 2015 @10:47AM (#10247697)
Re:AWS US East is experiencing high error rates on several s... by smcnally on Sunday September 20, 2015 @11:07AM (#10247776)
There is only Zuul?
Re:AWS US East is experiencing high error rates on several s... by clebio on Sunday September 20, 2015 @05:59PM (#10249340)
Re:AWS US East is experiencing high error rates on several s... by clebio on Sunday September 20, 2015 @05:57PM (#10249330)
Can you hint at what this means, for the uninitiated?
Re:AWS US East is experiencing high error rates on several s... by Tiksi on Sunday September 20, 2015 @08:51PM (#10249828)
I think he was making a reference to ghostbusters: http://knowyourmeme.com/memes/there-is-no-dana-only-zuul
Re:AWS US East is experiencing high error rates on several s... by georgeott on Sunday September 20, 2015 @10:48AM (#10247700)
Pocket just came back up also. Issue might be resolved?
Re:AWS US East is experiencing high error rates on several s... by ams6110 on Sunday September 20, 2015 @11:22AM (#10247839)
I was watching Netflix this morning and a number of titles had troubles loading repeatedly. They did eventually work though.
Re:AWS US East is experiencing high error rates on several s... by toomuchtodo on Sunday September 20, 2015 @10:44AM (#10247686)
Re:AWS US East is experiencing high error rates on several s... by stickfigure on Sunday September 20, 2015 @02:05PM (#10248389)
This is silly. Do you think you and your staff will maintain your homebrew RabbitMQ installation with better uptime than Amazon's team? And how much time/expense/manpower are you willing to devote to setting it up and monitoring it? And is that going to make your startup more successful than if you spent those resources developing the features your customers are asking for?
Re:AWS US East is experiencing high error rates on several s... by xbasic on Tuesday September 22, 2015 @07:13AM (#10257831)
Yes, competent people with good planning can easyly do a lot better than that.

We do serious hosting for government and company and if we had a outage like amazon had yesterday, we would loose at least half of our clients and pay heavy cash penality.

I don't understand why people give so much a break to amazon. They have one of the worst uptime.

Re:AWS US East is experiencing high error rates on several s... by faizalzakaria on Sunday September 20, 2015 @11:49AM (#10247938)
Kinda agree, same goes to AWS tools, Opsworks screwed us up back then due to AWS outage, and decided to decouple as much as possible from Opsworks.
Re:AWS US East is experiencing high error rates on several s... by jroid on Sunday September 20, 2015 @09:12AM (#10247374)
You still depend on some provider. Or, are you talking about multi cloud installations ?
Re:AWS US East is experiencing high error rates on several s... by rue on Sunday September 20, 2015 @09:20AM (#10247397)
Alternatively, run a fallback data store.
Re:AWS US East is experiencing high error rates on several s... by onetom on Sunday September 20, 2015 @10:26AM (#10247608)
I would highly recommend http://www.datomic.com as a database for multiple reasons, but one of them is that fact that it's totally portable between DynamoDB, PostgreSQL, MySQL, etc.

You can create a Datomic backup from any of these databases and restore them into a different one with the exact same semantics.

Re:AWS US East is experiencing high error rates on several s... by jsprogrammer on Sunday September 20, 2015 @10:46AM (#10247695)
Isn't datomic closed source?
Re:AWS US East is experiencing high error rates on several s... by onetom on Sunday September 20, 2015 @11:01AM (#10247743)
It is closed source, just like DynamoDB.

However - unlike DynamoDB - it has a quite usable free version and an even better starter version, which are good enough for production use. (The DDB local version is really only good for development) By production use, I mean for example a non-replicated MongoDB/PgSQL/MySQL, which is what I see from many smaller companies... They would be already really better of with Datomic.

If you really need more features, like high availability and transparent query caching with memcached, then the cheapest paid version has a one-time cost of $3000: http://www.datomic.com/pricing.html

You can easily save that amount of money by not spending 3-6 months developing tons of unnecessary queries, try to maintain that code and try to optimize its performance for scale...

On the "open-sourceness" note, I feel obliged to mention https://github.com/tonsky/datascript which IS open-source, and has a Datomic-like interface, but it's "just" an in-memory implementation.

But we are getting off-topic here I guess...

Re:AWS US East is experiencing high error rates on several s... by nadams on Sunday September 20, 2015 @10:57AM (#10247727)
That's what it looks like. And after 10 seconds of reading I couldn't tell what it actually does.