Re:Riemann – A network monitoring system
by bbrazil on Tuesday January 19, 2016 @10:01AM
(#10931151)
You need a place with all that information anyway, otherwise how do you alert on something being missing?
If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-m...
Re:Riemann – A network monitoring system
by zimpenfish on Tuesday January 19, 2016 @10:13AM
(#10931249)
If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.)
Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.
Re:Riemann – A network monitoring system
by bbrazil on Tuesday January 19, 2016 @10:47AM
(#10931453)
You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops.
Alerting based on state changes is fragile, it's better to compare against what you expect to see.
You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discov...
(Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)