Re:Systems Monitoring with Prometheus and Grafana
by sagichmal on Tuesday August 11, 2020 @06:42PM
(#24126852)
Prometheus "scales" really well, but it does so via segmentation and federation, rather than increasing the size of an e.g. cluster. Some use cases don't fit to that model, so projects like Cortex and Thanos exist.
[ Parent
| on HN ]
Re:Systems Monitoring with Prometheus and Grafana
by nielsole on Tuesday August 11, 2020 @06:57PM
(#24126948)
not vertically at least. the memory usage for indexing has room for improvement.
If I read the pprofs correctly, every scrape interval and every remote write allocates huge amounts of memory which is only cleaned up on garbage collection. You can easily need >64 gb ram for tenthousands of time series, otherwise you oom.
Re:Systems Monitoring with Prometheus and Grafana
by bjakubski on Tuesday August 11, 2020 @07:32PM
(#24127163)
Biggest single promethueus server I have access to currently uses almost 64GiB of RAM and ingests about 80000 samples per second. Most of scrape intervals is 60s.
It is about 5 000 000 time series.
Note that we do have more time series - above server is just a horizontal shard, ingesting just one part of total metrics volume there.
Re:Systems Monitoring with Prometheus and Grafana
by dikei on Tuesday August 11, 2020 @08:48PM
(#24127577)
Tens of thousands seems rather low, we are running 3 million series with less than 32GB of RAM and still have room to spare.