Systems Monitoring with Prometheus and Grafana

Re:Systems Monitoring with Prometheus and Grafana by nielsole on Tuesday August 11, 2020 @06:57PM (#24126948)

not vertically at least. the memory usage for indexing has room for improvement. If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by bjakubski on Tuesday August 11, 2020 @07:32PM (#24127163)

Biggest single promethueus server I have access to currently uses almost 64GiB of RAM and ingests about 80000 samples per second. Most of scrape intervals is 60s. It is about 5 000 000 time series. Note that we do have more time series - above server is just a horizontal shard, ingesting just one part of total metrics volume there.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by dikei on Tuesday August 11, 2020 @08:48PM (#24127577)

Tens of thousands seems rather low, we are running 3 million series with less than 32GB of RAM and still have room to spare.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by base698 on Tuesday August 11, 2020 @07:27PM (#24127130)

I do 15 million on about 64GB average memory. Have you tried recently?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by sagichmal on Tuesday August 11, 2020 @07:07PM (#24127014)

This has not been my experience at all. I'd file that as a bug.

[ Parent | on HN ]