I think AWS is early to offer a not-yet-matured product just to get it in front of customers and start gathering real usage feedback. I’d consider dynamo a fantastic tier 1 service, but it wasn’t always that way. This is inline with the Amazon philosophy in general.

As for “making something for the sake of having it”, I believe the key reason for building their own applications is that they can build it on the same multi-az incremental ledger/block store that underlies most services (first built for aurora), or a variant thereof. They have some pretty stellar underlying tech built to be reliable at cloud scale, something you won’t get from running a service on your own EC2 fleet. The new stuff built on top of it just isn’t all that mature though. That’s just my take though. If interested, the deep dive talk from reinvent last year on aurora is pretty cool.

Disclaimer: I work at Amazon, nothing to do with AWS or timestream. In fact, I’ve been excited for time stream for a while but found its initial performance lackluster for what I needed.

>I think AWS is early to offer a not-yet-matured product just to get it in front of customers and start gathering real usage feedback.

It's not just this. In many cases, the services launched are essentially MVPs that were only created to serve a handful of customers' needs. If a huge whale of a customer says they want hosted Jupyter notebooks on AWS (just a theoretical example I picked because I saw someone criticize Sagemaker, but I don't have any specific knowledge of Sagemaker), then AWS will create that for them, but, AWS doesn't actually do "one off" services for a specific customer, so they will also launch those hosted notebooks as a public service.

The result is of course that this service was built with one customer's use case in mind, so it might not serve other customers' use cases very well, which leads to some negative perception. But ideally AWS then goes and incorporates feedback into the service, turning it into a "tier 1" service rather than an MVP. It just takes time.

I used to work for one of AWS's biggest customers. This was 100% how we operated, we told AWS what we wanted them to solve, and they would build it for us. When we're done closed beta testing it and getting changes in, it gets released as a public service a year later or so.

They are also incentivized to do it because it makes AWS sticky. The more of their services you use, the harder it is to migrate somewhere else. They don't need to put out quality services, they just have to put out enough that you become dependent on one or two of them and then you're stuck.

Which deepdive talk are you recommending?

AWS re:Invent 2019: [REPEAT 1] Amazon Aurora storage demystified: How it all works (DAT309-R1): https://www.youtube.com/watch?v=DrtwAOND1Pc

But all of the Deep Dive talks are good for technical details: https://www.youtube.com/results?search_query=aws+reinvent+20...

I remember the announcement of Sagemaker at an AWS summit, a moment of profound embarrassment - the whole thing was barely a Jupyter notebook glued to some EC2 stuff with duct tape.

You're right, they're definitely doing things for the sole purpose of locking you in.

I still do not understand any of the value add of Sagemaker.

...have you looked at it recently? It's a MASSIVE suite of services, tools, and capabilities...https://aws.amazon.com/sagemaker/

You still have to compile your own Docker image and then "run it" from a Jupyter notebook if you are not using their Estimators (who uses those?). What is the point of just launching Docker images from Jupyter notebooks and looking a different window to see the log?

Serving is ok on Sagemaker, i will admit. The new ones: feature store, pipelines, and data wrangler - we will see, all rushed (some more than others).

Working at AWS must be fun

You basically reinvent all the stuff - from databases to all avaliable tooling

normal companies dont do that, so there' an opportunity to get deep into some specific branch of applied informatics.

They even name their annual event this way[1]

[1] https://reinvent.awsevents.com

I think, the difference is that AWS already has a bunch of core services that scale and uses them as a base for new services. (MongoDB->DocumentDB)

Also, they hear about the problems some customers have with an existing product and try to (re)invent a better solution. (MySQL->Aurora)

Sometimes I'm not even sure they're re-inventing, just adding a vender-lock-in layer on top of the service and making it run on Lambda so they can advertise it as "web-scale".

Yeah, although not on the AGPL licensed services.

It’s fun if you’re the guy building the stuff. Not if you’re the guy using all reinvented tooling.

But why do they do it so badly? Seems like they have the engineering muscle to put out a great reinvention of a document db, a time series db, a graph db... but each is relatively poor to the other services on AWS. Is it the combination of MVP-culture and lock-in?

The article suggests an answer:

"The viability of our company, Timescale, is 100% dependent on the quality of TimescaleDB. If we build a sub-par product, we cease to exist. Amazon Timestream is just another of the 200+ services that Amazon is developing. Regardless of the quality of Amazon Timestream, that team will still be supported by the rest of Amazon’s business – and if the product gets shut down, that team will find homes elsewhere within the larger company."

Breadth versus depth seems to be the play outside of their absolute core offerings. It allows people to dip their toes into something with minimal commitment and by the time they realize the shortcomings they might conclude that migrating for those features just isn't worth it.

So this give them a bit of lock-in. Come for the EC2, stay for the minimally viable queues, machine learning, containers, etc.

This exactly, they are just trying to get their foot in the door in literally any idea someone else is already pursuing so that if it starts to make money they can start to put the juice on the engineering for that and take their market share.

The business model is solely focused on competing with and taking out other businesses, not actually providing any value. AWS is like a pack of sharks.

The value they sell is really operational convenience. Over time they have figured out how to run large services at scale, and monetize that. The features of those services are often weaker than competition, as noted in this thread.

There are only so many "A" teams in a big company.

it would be more fun without the annual reinvent death march

> ...see here Kinesis instead of just doing Kafka and many other products

Kinesis was a solution to their in-house woes with metering and billing which absolutely drowned in so much data from internal services with myriad metering and billing items and rules that they had to do something about it themselves: https://gigaom.com/2014/03/20/why-amazon-built-its-data-stre...

BTW, Kinesis Data Firehose is a pretty good product.

Kinesis Data Firehose is now pretty much Flink. Flink is a pretty good product.

It seems like AWS is just focusing on increasing it's vendor lock in and it makes sense not just from the AWS point of view but if you're a developer at an enterprise company that has authorised AWS the more databases the clone, poorly or badly the more choice you have without having to get authorisation and the cost doesn't matter to the developer since the enterprise company is paying for it.

I suspect there are many enterprises that are already so locked into AWS that it would take them a decade to move to a different provider.

I don't have much expertise with Timestream. Can't comment on it. But, there is nothing wrong in trying for an alternate implementation. We're using Kinesis and it is a good fit in our stack with lot less overhead. They did bring Kakfa (Amazon MSK) later to the AWS service list. Its upto the consumer to pick Kinesis or MSK.

One example which was influential in nature - Dyanmodb. The Dyanmodb paper inspired a number of NoSQL databases, including Apache Cassandra. If they have went with the popular NoSQL database at that time, we might have missed some good contribution.

But you can just run timescaleDB inside your amazon VPC. No need to use their services at all.

You can but then you need to spend time setting up and maintaining the service. Which many enterprises aren't willing to do. Also, while you may be authorised to use AWS tech you may not be automatically authorised to use different tech. We literally were told to use MySQL instead of MongoDB soley because RDS was allowed and all services had to be managed by AWS.

I am not complaining AWS is doing vendor lock in makes complete sense, I am just commenting that is their main direction for their offering. And their vendor lock is aimed at the enterprises where they can really run wild on it and get lots of money in return. Smaller orginisations will generally never lock themselves in so much because they won't build so many things that are so hard coupled since they just don't have the manpower.

And we in fact have fully open-source/free k8s helm charts to easily spin up TimescaleDB, including with HA and automated failure detection/failover and streaming replication for incremental backups:

https://github.com/timescale/timescaledb-kubernetes

Or just use Timescale's managed cloud on AWS:

https://www.timescale.com/timescale-signup

Much like Amazon themselves with breaking away from using Oracle.

In a recent webinar for capital markets, AWS didn't even try to advertise Timestream, they talked about the three most popular tsdb in finance and that's it.

Source: https://pages.awscloud.com/awsmp-h2-fin-time-series-database...

And those 3 are?

OneTick, Kx, and QuasarDB

Thanks!

In my opinion it would be very hard to justify using Timestream for any analysis heavy workloads for at least three reasons:

1. Queries will need to touch a lot of data - will cost a lot, and there is no ability to optimize the queries in any way (no EXPLAIN, no indexes, no downsampling)

2. No integration with data exploration and visualization tools

3. No ability to have non-timeseries data, or correlate any data in two different tables (no JOINs)

(Disclaimer: I work at TimescaleDB)

I completely agree I think you misunderstood me.

I would like to see comparisons between Timescale and Druid.

I already know Timestream isn't up to the task lol.

What about comparing TimescaleDB to VictoriaMetrics? There are some old benchmarks [1], but it would be great to see updated benchmarks as well for the latest TimescaleDB and VictoriaMetrics versions.

[1] https://valyala.medium.com/measuring-vertical-scalability-fo...

What kind of auth & logging does TimescaleDB offer? Is every request authenticated & authorized?

TimescaleDB inherits from Postgres a rich role-based access control, so all requests require permissions and roles can be enforced at various levels (database, schema, table, or even rows) and privileges (SELECT, INSERT, DELETE, CREATE, etc).

Authentication to the database is typically governed to different security mechanisms, including certificate-based auth, password via SSL, etc.

How to tell what service is in what tier, crap or good? What's the metric? I am not a direct user of AWS, so absent an informed technical judgment of "this is a deployment and renaming of that", eg Kinesis for Kafka, how do you know what to avoid? In general, it seems like the value-add for service X is that now that function is integrated to some degree w/ the rest of AWS - so maybe it's based on effective feature set/use cases plus integration points. Alternatively, some function of throughput and unit cost.

If I were a SAAS competing with an AWS service, I would want to know how long before a given AWS service might get good enough to be an alternative for my customers. If you run with the idea that AWS offerings get better over time and eventually cross some threshold of "not good" to "good", maybe a crawl of the AWS announcements[0] in combination with a metric could provide a predictor of an AWS service's time to "good"...

[0] https://aws.amazon.com/new/