About 2/3 just came down to filtering certain classes of errors. 4xx errors, I stopped email notification altogether, since they were already being trapped/handled by the system. Others were a little more specific. Ironically, .Net tends to handle some things that should be 4xx errors as 500, so reclassifying those took out a lot as well.
In the end, within about a month, the emails were down to a manageable 20 or so a day and got more visibility as a result.
Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it’s simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It’s ultimately an operational problem, not a technical one.
Alas, back in the real world, logging is always the last thing teams have time to think about...
I’ve also had clients in the past use Splunk with ML forecasting models that inject fields as part of the ingest pipeline. I don’t know the details of that implementation; I just know how the dev teams were using it.
In your example, the `NumberFormatException` is a bad ERROR entry, because it's covered by a WARN entry right below it. Meaning that the request was not failed -- it entered a default value instead of the bad parsing. So that exception should also be at WARN level.
(Arguably, overwriting input values with defaults because of a parsing error is probably a bad idea and should be an ERROR due to rejecting the request. But I'm rolling with what we got here.)
I write apps that generate lots of logs too...I think an improvement lies in some form of automated algorithmic/machine learning (to incorporate a buzzword in your pitch) log analysis.
When I page through the log in a text editor, or watch `tail` if it's live, there's a lot of stuff that looks like
Visually (or through regex), you can filter out all the "Request initiated" noise. Maybe the default value warning occurs 10% of the time, and is usually accompanied by that number format exception (which somebody should address, but it still functions, and there's other stuff to fix). But maybe the "Timeout error" hasn't been seen in weeks, and the value of C has always been positive - that is useful information!Don't email me when there's a profile incomplete warning. Don't email me any time there's an "ERROR" entry, because that just makes people reluctant to use error level logging. Definitely don't email me when there's a unique request complete string, that's trivially different every time. But do let me know when something weird is going on!