Re:How to lose $172k per second for 45 minutes (2013)
by mattdodge on Monday April 01, 2019 @12:22PM
(#19544983)
Care to share what types of tools do this? I'm genuinely interested. I haven't come across a log management tool that uses AI to detect abnormal conditions based on the log message contents like the OP describes. I stick to Papertrail for the most part though so I'm likely out of the loop.
Re:How to lose $172k per second for 45 minutes (2013)
by exelius on Monday April 01, 2019 @12:35PM
(#19545140)
I’ve used and really liked DataDog in the past. It has some rudimentary ML functionality for anomaly detection of certain fields, but it’s only getting better.
I’ve also had clients in the past use Splunk with ML forecasting models that inject fields as part of the ingest pipeline. I don’t know the details of that implementation; I just know how the dev teams were using it.
Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it’s simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It’s ultimately an operational problem, not a technical one.
Alas, back in the real world, logging is always the last thing teams have time to think about...