Systems Monitoring with Prometheus and Grafana

Re:Systems Monitoring with Prometheus and Grafana by ecnahc515 on Tuesday August 11, 2020 @08:19PM (#24127427)

> Erything about the design (polling, single database) tells me that it was designed primarily to sit alongside something small.

Prometheus is designed to be "functionally sharded". You shouldn't be running one "mega prometheus". Often it's something like 1 Prometheus per-team, depending on the amount of metrics each produces.

You can use federation at lower resolutions or a one of the distributed setups (Thanos/Cortex) if you want to avoid extra storage or lower resolution that federation entails.

> But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives

Not to distract, but I think you meant resolution, not cardinality. Cardinality is the metadata like labels/dimensions. Resolution is the granularity in the time.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by AlphaSite on Wednesday August 12, 2020 @01:01AM (#24128733)

Doesn’t it also have cardinality issues?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by halfmatthalfcat on Wednesday August 12, 2020 @01:30AM (#24128852)

You get high enough (hitting a metric with >100k unique labels), queries become unmanageable and incredibly slow when backed by the stock datastore (tsdb). However there are backing datastores (TimescaleDB, InfluxDB, VictoriaMetrics, etc) that ingest Prometheus metrics and enable higher cardinality.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by unixhero on Wednesday August 12, 2020 @08:05AM (#24131177)

This is where I fall off.

Is Prometheus a DB that (can) forward data to another DB?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by halfmatthalfcat on Wednesday August 12, 2020 @08:32AM (#24131367)

Prometheus is a data format but it's also a "suite" of tools on top of that data format.

Usually what happens is your app, db, whatever will expose metrics (http request status, average response time, etc) in the Prometheus format which is then scraped by the Prometheus ingestor. The ingestor stores those metrics in a (short-term) datastore called TSDB. Prometheus also ships with a little web UI as well that can query those metrics in TSDB.

However Prometheus allows scraping from the ingestor (usually federation) or pushing into an external datastore that is usually more performant than TSDB.

So when people say "Prometheus", they usually are talking about the suite of tools however practically it's really just the format of the metrics data.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by unixhero on Thursday August 13, 2020 @03:24PM (#24147285)

Thanks for the explanation

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by ecnahc515 on Wednesday August 12, 2020 @06:53PM (#24138028)

Sort of. https://prometheus.io/docs/prometheus/latest/storage/#remote...

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by roskilli on Wednesday August 12, 2020 @02:17AM (#24129069)

And M3DB too if you want to cluster and scale out, vs cloud store

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by RhodesianHunter on Wednesday August 12, 2020 @01:18AM (#24128811)

There's no real way to develop a metrics system without cardinality issues. Where you draw the line depends on the backing database but they're all fairly constrained.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by ashtonkem on Wednesday August 12, 2020 @02:22AM (#24129094)

Honestly, it’s incredibly impressive how far you can push them as is. We send a lot of data into these systems.

[ Parent | on HN ]