FWIW the company behind Elasticsearch is happy to sell you a solution for that: https://www.elastic.co/products/watcher

I am using it right now in production. It's worst trait is that it is not clustered and does not share state about what alerts have been sent already. This means you can have no more than one agent running without having duplication of alerts. This also complicates high-availability for obvious reasons.

(Correction/Public Shaming: Its)

My biggest issue with this was that it is purely event driven. So what if your trying to alert on no event? They had no answer for that

Right. This is something you can get quite easily using Riemann.

I have been running this. It works well. I also wrote some custom code to send the stack-traces, if we get any, to a slack channel.

That sounds fantastic. Do you have an example of how you extracted the trace? Was it simple to extend ElastAlert or did you have to fork it to add that kind of option?

The trace is basically extracted using Logstash's multiline filter. It adds a tag "stacktrace" to the log entry.

I just made a few changes in SlackAlerter class to customize for my use-case. I'd try to clean-up and send a PR soon.