News for hackers. Stuff that matters.
Re:Monitoring Microservices with Prometheus and Grafana by Raed667 on Friday December 09, 2022 @05:10AM (#33919656)
If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/

Also sometimes you'll have some sort of a batch job, or a cron that can't be scrapped the regular way, checkout Prometheus Pushgateway https://prometheus.io/docs/practices/pushing/

Re:Monitoring Microservices with Prometheus and Grafana by ilyt on Friday December 09, 2022 @06:16AM (#33920101)
I really don't get why "scrape the prometheus endpoint" is a go-to now, push model seems to be way less PITA to manage at scale.

> If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/

Any idea how it compares with https://victoriametrics.com/ ?

We're slowly looking for a replacement for InfluxDB (as 1.8 is essentially on life support), the low disk footprint is pretty big advantage here.

Re:Monitoring Microservices with Prometheus and Grafana by buro9 on Friday December 09, 2022 @06:56AM (#33920305)
Scrape is typically just how people get started and works well for small and medium things, it gets you a long way before you need to consider it.

Prometheus remote_write is what people graduate to, this gets you the rest of the way, and you are correct it's less PITA at scale.

If you're looking for retention your choices are large, there's Cortex (CNCF), Mimir (most Cortex work moved here), Thanos, VictoriaMetrics, TimeScale, Chronosphere, and many others.

All seek to do a similar thing from a distance, they all store metrics (likely from Prometheus) and allow retention and some variety of how to query it (if you want SQL you got it, if you want non-standard functions you go it, if your reads are more important than your writes you got it, if you need a billion active series you got it, etc).

If what you want is "Prometheus but bigger" then the Prometheus project provides a compliance suite that you can run to help you evaluate your options: https://github.com/prometheus/compliance

I work for Grafana Labs, and we have maintainers working for us who have touched Prometheus, Thanos, Cortex and Mimir. Mimir is currently the largest investment we have https://github.com/grafana/mimir and it is 100% compliant with Prometheus (though that is about to be temporarily untrue as Native Histograms is landing in Prometheus soon https://github.com/prometheus/prometheus/milestone/10 and we'll need to add a perfectly compliant support to Mimir to get back to being compliant).

Re:Monitoring Microservices with Prometheus and Grafana by Rapzid on Friday December 09, 2022 @08:10AM (#33920822)
Interesting... I recall the prom creators being very, very convinced that pull was the one true way (push gateway was viewed as a compatibility concession).
Re:Monitoring Microservices with Prometheus and Grafana by capableweb on Friday December 09, 2022 @11:33AM (#33923220)
I mean, in most cases it just works, for typical "Here is my HTTP server" services. But for batch processes, not so well, as you're not exposing a HTTP endpoint.

Pull does work like 90% of the time, + you get a handy uptime check built-in as well, as if the pull didn't work, the service is probably offline and you'll notice it immediately. With Push you still need a way of checking if the service is online/offline.

Re:Monitoring Microservices with Prometheus and Grafana by oofnik on Friday December 09, 2022 @09:58AM (#33921867)
We began a transition from a single behemoth Prometheus instance to Victoria Metrics about a month ago after evaluating a couple of similar tools. VM won out because of its relative operational simplicity and eye-popping benchmark results in resource usage comparisons. Documentation can be a little rough around the edges, but overall we're impressed so far.
Re:Monitoring Microservices with Prometheus and Grafana by heliodor on Friday December 09, 2022 @08:25AM (#33920958)
VictoriaMetrics is super simple to use and reliable. It's a breath of fresh air compared to running other tools.
Re:Monitoring Microservices with Prometheus and Grafana by Raed667 on Friday December 09, 2022 @06:46AM (#33920256)
> push model seems to be way less PITA to manage

They have a blog post explaining their reasoning:

https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or...

Re:Monitoring Microservices with Prometheus and Grafana by cduzz on Friday December 09, 2022 @07:21AM (#33920469)
"Hey, I'm trying to figure out some business metrics related to our infrastructure and application performance / uptime; do we have the data anywhere?"

"Sure, Bob, just write a promql query against this interface"

https://news.ycombinator.com/item?id=1636348

(in case my point is too subtle: I believe that the value of long term metric retention is to support business decisions more than supporting availability related monitoring.

The tools "business deciders" use to look at "the things they call metrics" are simply unable to look at data lodged in a prometheus silo. So the data you have for long term retention is simply unusable for the thing it's most valuable)

(edited for clarity)

Re:Monitoring Microservices with Prometheus and Grafana by flerchin on Friday December 09, 2022 @08:44AM (#33921125)
I use thanos for availability related concerns. eg, Some metric crossed the threshold for saturation, and I can go back as far as I need to see when the metric began saturating. We have seasonal business patterns that I can show the high water-mark for resource consumption. Thanos also allows a CQRS-type pattern if that's important to folks.

Do I point the biz at it? Only when I'm trying to show them availability related metrics.

Re:Monitoring Microservices with Prometheus and Grafana by gjulianm on Friday December 09, 2022 @08:54AM (#33921217)
What tools do they use? It's not hard to build a dashboard in Grafana and show them there the metrics they want, in an easily-digestible format.
Re:Monitoring Microservices with Prometheus and Grafana by cduzz on Friday December 09, 2022 @11:37AM (#33923285)
There's an entire universe of people making business decisions who live in excel and "BI" tools. If your thing doesn't speak ODBC / JDBC it's invisible.

Further, the vast majority of workaday devs are stressed out and under tremendous pressure to thing all the things in their ticket queue. They're probably pretty fluent in their thinging language and likely also conversant in SQL. Asking them "hey learn this other query language, because" is just adding stress to them. Sure, some are going to rise above and promql things too, but for lots, it's telling them to "write a distributed map reduce query in erlang."

Re:Monitoring Microservices with Prometheus and Grafana by gjulianm on Saturday December 10, 2022 @07:28AM (#33932019)
PromQL is barely a language. For availability, the query would be as simple as the series name and it would return everything. Also, there's an adapter so that Prometheus writes to PostgreSQL.

However, Prometheus is not a tool that specializes in reporting availability to people who live in Excel. Prometheus scrapes metric from a lot of systems in a standard format, and enables you to see what's happening and write alerts to fix things when they break, without requiring complex frameworks. Reporting availability to people is just a small, tiny thing that Prometheus is able to do.

Re:Monitoring Microservices with Prometheus and Grafana by marginalia_nu on Friday December 09, 2022 @06:56AM (#33920304)
Batch jobs are arguably a lot better served with old-school logging than metrics. If you're using prometheus for alerts, I'll maybe go as far as considering incrementing a "the batch job failed"-counter a good idea. But all this is very much jamming a square peg into a round hole.
Re:Monitoring Microservices with Prometheus and Grafana by encoderer on Friday December 09, 2022 @11:48AM (#33923433)
Also, there are tools like Cronitor that can help.
Re:Monitoring Microservices with Prometheus and Grafana by packetslave on Friday December 09, 2022 @01:33PM (#33924821)
I would highly recommend looking at Cortex (or Grafana Labs' fork, Mimir) instead of Thanos.

It doesn't rely on reading the Prometheus WAL (rather, it uses Prometheus's built-in remote-write functionality). This is both less fragile and has the benefit of being about to run Prometheus in stateless mode (or replace it entirely with something like Grafana Agent, vmagent, etc.

Grafana Labs has done a proof of concept showing Mimir scaling to 1 billion active timeseries. There are also "real" users serving 700+ million metrics in production.

Re:Monitoring Microservices with Prometheus and Grafana by saifulwebid on Sunday December 11, 2022 @12:04AM (#33940296)
How do you “import” existing Prometheus metrics block into Cortex/Mimir using the remote-write, then?
Re:Monitoring Microservices with Prometheus and Grafana by packetslave on Thursday December 15, 2022 @12:19PM (#34002690)
For Thanos, you can (mostly) just copy the blocks from the existing s3 bucket into Mimir's bucket. There's some cleanup prep around external labels, but nothing too difficult, and the process is well-documented. I just finished migrating a year of Thanos metrics into Mimir without issue.