News for hackers. Stuff that matters.
Re:Riemann – A network monitoring system by MayMuncher on Tuesday January 19, 2016 @09:27AM (#10930911)
I don't like it since the master basically does a curl on each server to get the info, so it doesn't work behind a firewall without tons of issues
Re:Riemann – A network monitoring system by zimpenfish on Tuesday January 19, 2016 @09:35AM (#10930958)
That's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector.

(Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)

Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @10:01AM (#10931151)
You need a place with all that information anyway, otherwise how do you alert on something being missing?

If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-m...

Re:Riemann – A network monitoring system by zimpenfish on Tuesday January 19, 2016 @10:13AM (#10931249)
If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.)

Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.

Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @10:47AM (#10931453)
You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops.

Alerting based on state changes is fragile, it's better to compare against what you expect to see.

Re:Riemann – A network monitoring system by vruiz on Tuesday January 19, 2016 @11:34AM (#10931769)
You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discov...
Re:Riemann – A network monitoring system by robzienert on Tuesday January 19, 2016 @09:37AM (#10930977)
That's what the Prometheus push gateway is for. While prom itself is poll-based, you can extract data from servers behind a firewall by pushing metrics from an agent into the gateway. No crazy issues.
Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @10:01AM (#10931157)
The general advice is to run Prometheus behind the firewall, you want as few things between your monitoring and what you're monitoring as possible. This makes it more resilient.
Re:Riemann – A network monitoring system by lobster_johnson on Tuesday January 19, 2016 @06:43PM (#10935004)
It supports push, though.