Re:Prometheus: Monitoring for the next generation of cluster...
by ymse on Sunday May 01, 2016 @06:58AM
(#11605727)
That's sensible, thanks.
What if the inside of the network is further segregated, and the Prometheus instance does not have access to all endpoints. Can I use the push gateway as a "proxy", or is a direct route required.
Sorry for the stupid questions. Now it's getting entirely academic of course :)
Edit: just re-read your post and saw the proxy note. Time to read the documentation again. This setup should make for a good blog post.
Re:Prometheus: Monitoring for the next generation of cluster...
by bbrazil on Sunday May 01, 2016 @08:42AM
(#11605965)
It's advised not to use the pushgateway in that fashion, it's for service-level batch jobs - not trying to subvert your organization's network security policies. See https://prometheus.io/docs/practices/pushing/
Re:Prometheus: Monitoring for the next generation of cluster...
by bogomipz on Sunday May 01, 2016 @11:55AM
(#11606651)
Can you monitor alerts via a federated scraping? So for example you have one master alerts dashboard?
Re:Prometheus: Monitoring for the next generation of cluster...
by XorNot on Sunday May 01, 2016 @05:34PM
(#11607957)
Alerts are push anyway at the moment - so sending multiple Prometheus alerts to a single alertmanager is easy.
But yes, the actual "alert firing" metrics are exposed by the federated endpoints.
Re:Prometheus: Monitoring for the next generation of cluster...
by bbrazil on Sunday May 01, 2016 @06:38PM
(#11608173)
You'd lose annotations and other metadata doing that, plus it's another hop on the critical path of alerting. Probably not the wisest of ideas. You want your alerts coming from as close to what you're monitoring as possible.
Though it is an odd requirement to need to only have Prometheus outside a firewall.
Edit: guessing from your post - the way I'd do it is run a Prometheus instance at all 4 sites, and have them all federate each other. That way each site is the HA redundancy for the others.