I've commented fairly heavily in the related Grafana thread.

Prometheus is a bit of a different story. It does have some operational overhead when you get to a certain point, and scaling it out is not always trivial.

Assuming it works, there is value-add on this one, and the pricing is more in line with active use (ie, a cost+ model, which is more typical of AWS services)

This seems more interesting of the two, grafana is pretty simple to setup and maintain. The harder part is handling the metrics themselves, be it with influxdb, prometheus, or something else.

Scaling prometheus across multiple separate Kubernetes clusters is a fking nightmare.

Use Victoria metrics. One lightweight agent per cluster pushing to a centralised metrics store makes it so much easier.

setting up one Prometheus server is easy. scaling, HA, Metrics retention for more than 3 days not so much.

Prometheus is not easy to run at scale on the storage side.

This is all relative but I don't personally think so. Not on EC2+EBS, anyway. Certainly not as difficult as running/scaling an ES or Kafka cluster.

It's a completely different problem because by default Prometheus does not shard anything so you're bound to a single instance, where ES and Kafka are cluster based.

Out of interest what do you find hard about running ElasticSearch clusters?

In my experience ES has been one of the easiest clustered / highly available and sharded systems I've ever run - especially for how incredibly performant and reliable it is.

I've generally found that beyond right sizing your nodes, indexes and shard configuration - it pretty much just works without ever really having issues.

Victoria Metrics is an absolutely superb drop in replacement.

It's not a drop-in replacement (even though it tries to sell itself as such), it's incompatible in a significant number of ways and throws away part of your data.

We use Victoria Metrics in Prod for more than 6 months. It is very reliable and scalable. Victoria metrics handles more than 2B+ series in our setup without breaking a sweat.

No, it's better. You can focus on the thing you're measuring rather than the bloody platform.

They made some pragmatic optimizations by dropping part of the data where it mostly doesn't matter. Who's actually affected by this? 0.01% of users?

You could say the same about any SaaS based on open source, but people still find it useful