News for hackers. Stuff that matters.
Re:Monitoring your own infrastructure using Grafana, InfluxD... by lima on Tuesday July 21, 2020 @10:42AM (#23907149)
Please elaborate - how is it one engineer's full time job?

We run Prometheus in production and this hasn't been our experience at all.

A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).

Re:Monitoring your own infrastructure using Grafana, InfluxD... by gen220 on Tuesday July 21, 2020 @11:21AM (#23907507)
I think the parent's complaint is a function of how your engineering org "uses" prometheus.

If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus).

IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".

Re:Monitoring your own infrastructure using Grafana, InfluxD... by dharmab on Tuesday July 21, 2020 @12:49PM (#23908442)
I'd agree; I'm at a large corp that has a need to store our data for a very long term. If we were using Prometheus as an ephemeral/short term TSDB to drive alerting only, it would be really easy.
Re:Monitoring your own infrastructure using Grafana, InfluxD... by user5994461 on Tuesday July 21, 2020 @02:13PM (#23909275)
For reference: A single machine can push 5k metrics so you're saying a single prometheus instance can easily serve 20s of hosts. lol