Re:Monitoring your own infrastructure using Grafana, InfluxD...
by gen220 on Tuesday July 21, 2020 @11:21AM
(#23907507)
I think the parent's complaint is a function of how your engineering org "uses" prometheus.
If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus).
IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".
Re:Monitoring your own infrastructure using Grafana, InfluxD...
by dharmab on Tuesday July 21, 2020 @12:49PM
(#23908442)
I'd agree; I'm at a large corp that has a need to store our data for a very long term. If we were using Prometheus as an ephemeral/short term TSDB to drive alerting only, it would be really easy.
Re:Monitoring your own infrastructure using Grafana, InfluxD...
by user5994461 on Tuesday July 21, 2020 @02:13PM
(#23909275)
For reference: A single machine can push 5k metrics so you're saying a single prometheus instance can easily serve 20s of hosts. lol
We run Prometheus in production and this hasn't been our experience at all.
A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).