Aurora and Redshift do not scale well enough for companies at their scale (only 64TB for Aurora last I checked, and Redshift falls over near the 100TB level). They'd be looking at DynamoDB.

I don’t think any managed AWS service scales well for any bigger enterprise which have to handle a decent amount of traffic. Once you reach certain threshold you soon hit some throttling and limits BS and AWS solution is just to throw more money at them.

DynamoDB scales a lot. It falls into the "throw money at it," bucket though, and you need to have a pattern that shards well with it. If you have a reason they will basically always increase your limits, too.

> Redshift falls over near the 100TB level.

This is based on what? They have stated in various presentations that they have customers using 1pb clusters.

In my personal experience.

There's a re:invent presentation of them having an in-house 1pb+ cluster. What they don't tell you is that 101 8xl nodes is half a million dollars a year in reserved instances, before you include any other costs associated with it; and that particular workflow (log scanning) is very nicely suited for any columnar store.

Operations are also a disaster with Redshift; anything where you have to touch the cluster itself at any scale past a handful of nodes typically requires a support ticket with "hey, when this breaks, please work your magic and fix it?" There's also the issue of tuning your queues, which is a whole extra layer that you, the customer, must tune. Their suggestion tools are getting better on that front though.

Just use something else if you have more than a few TB, or you have a ton of time and money to just throw around.

Ha, $500k/year is the amount a salesperson can knock off the cost of Teradata 100TB setup without calling their boss.

As for ops stuff, I guess YMMV. We’ve never had any significant issues. Tuning is the same or less than on-prem systems IME.

Perf is still pretty much unmatched in the cloud, especially perf/$. Try to do a join on 100tb in a competing system (e.g. BQ) and it won’t finish.

Out of curiosity, is the binlog accessible on Aurora/MySQL? We've lots of users of the Debezium MySQL connector on AWS, but would be interesting to know whether it'd work with Aurora, too.

(Disclaimer: I'm the lead of Debezium, which is the open-source CDC platform to which WePay are contributing their Cassandra work)