Systems Monitoring with Prometheus and Grafana

Re:Systems Monitoring with Prometheus and Grafana by cmckn on Tuesday August 11, 2020 @06:29PM (#24126752)

Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had access to that telemetry before. From the existence of Cortex[1], it seems that Prometheus doesn't scale incredibly well, but I don't think many users will hit these limits.

[1] https://cortexmetrics.io/

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by edoceo on Tuesday August 11, 2020 @06:37PM (#24126815)

My Prometheus system is a $10/mo Linode. It collects from 27 other hosts, and at least 100 services distributed across those hosts - doesn't even break a sweat. All the exporters run through a wireguard VPN. Prometheus is great for a small/medium SaaS type environment.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by abhishekjha on Wednesday August 12, 2020 @01:18AM (#24128806)

What do you use as a frontend? As far as I could tell grafana free tier doesn’t allow monitoring cluster of servers.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by edoceo on Wednesday August 12, 2020 @10:19PM (#24139324)

I use Grafana and some custom ones, I have only one Prometheus box so clustering is not a problem I'm having (and likely won't, I can vertical scale a long way for my smallish operation)

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by dewey on Wednesday August 12, 2020 @07:21AM (#24130839)

You could self host it.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by abhishekjha on Wednesday August 12, 2020 @09:21AM (#24131786)

Can I self host for monitoring cluster of servers? Currently I have grafana installed on each of my servers and I am having to monitor them individually. I want a centralised dashboard over telegraf + influxdb.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by detaro on Wednesday August 12, 2020 @09:22AM (#24131799)

Why would you install Grafana + Influx on each server instead of one central one?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by abhishekjha on Wednesday August 12, 2020 @09:51AM (#24132066)

I haven't spent much time on this but most of the docs were for setting it up on each hosts. Is there a proper tutorial for clusters?

Also I wanted to keep the monitoring unaffected for other servers if one of them go down. If I setup a central server for monitoring then that becomes a single point of failure.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by heliodor on Wednesday August 12, 2020 @05:54PM (#24137462)

Grafana is meant to run as a single instance. For monitoring multiple servers, you need to get the metrics into one data store, from which Grafana will read. That's Prometheus' job. These pieces should not be on the same servers that run your product. For HA, you can run two or more Prometheuses as duplicates, so you can switch to another one if the main one is down.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by abhishekjha on Wednesday August 12, 2020 @08:38PM (#24138794)

Would the one datasource have a single database with several tables, one for each server? Lets say I am monitoring mysql. Currently I have a `mysql` table in a databse named `telegraf` on each host. Can I combine multiple influx datasources into a single dashboard beacause that would be easiler right now for my current setup?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by heliodor on Thursday August 13, 2020 @07:07AM (#24142040)

You handle multiple servers by tagging your metrics with the server id or name. No need to create a table per server.

Each panel in a dashboard points to a specific data source, so you can have multiple data sources in one dashboard.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by sagichmal on Tuesday August 11, 2020 @06:42PM (#24126852)

Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by nielsole on Tuesday August 11, 2020 @06:57PM (#24126948)

not vertically at least. the memory usage for indexing has room for improvement. If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by bjakubski on Tuesday August 11, 2020 @07:32PM (#24127163)

Biggest single promethueus server I have access to currently uses almost 64GiB of RAM and ingests about 80000 samples per second. Most of scrape intervals is 60s. It is about 5 000 000 time series. Note that we do have more time series - above server is just a horizontal shard, ingesting just one part of total metrics volume there.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by dikei on Tuesday August 11, 2020 @08:48PM (#24127577)

Tens of thousands seems rather low, we are running 3 million series with less than 32GB of RAM and still have room to spare.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by base698 on Tuesday August 11, 2020 @07:27PM (#24127130)

I do 15 million on about 64GB average memory. Have you tried recently?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by sagichmal on Tuesday August 11, 2020 @07:07PM (#24127014)

This has not been my experience at all. I'd file that as a bug.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by jeffbee on Tuesday August 11, 2020 @11:47PM (#24128430)

Interesting to say that Prometheus is "fledgling". The project is almost 7 years old and the Google thing on which it is based is ~15 years old.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by cmckn on Wednesday August 12, 2020 @03:19AM (#24129406)

> I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling...

I didn’t know the age of the project, because I hadn’t heard of it. That’s why I go on to say that in actuality it has a ton of adoption and I’ve had a great experience with it.

[ Parent | on HN ]