> I came away from this thinking the worst part of this was that their system did send them errors, its just that when you deal with billions of events emailing errors just tend to get ignored as at that scale you generate so many false positives with logging.

I write apps that generate lots of logs too...I think an improvement lies in some form of automated algorithmic/machine learning (to incorporate a buzzword in your pitch) log analysis.

When I page through the log in a text editor, or watch `tail` if it's live, there's a lot of stuff that looks like

    TRACE: 2019-04-01 09:45:03 ID A1D65F19: Request 1234 initiated
    ERROR: 2019-04-01 09:45:04 ID A1D65F19: NumberFormatException: '' is not a valid number in ProfileParser, line 127
    WARN : 2019-04-01 09:45:04 ID A1D65F19: Profile incomplete, default values used
    WARN : 2019-04-01 09:45:14 ID A1D65F19: Timeout: Service did not respond within 10 seconds
    TRACE: 2019-04-01 09:45:14 ID A1D65F19: Request 1234 completed. A = 187263, B = 1.8423, C = $-85.12, T = 11.15s

Visually (or through regex), you can filter out all the "Request initiated" noise. Maybe the default value warning occurs 10% of the time, and is usually accompanied by that number format exception (which somebody should address, but it still functions, and there's other stuff to fix). But maybe the "Timeout error" hasn't been seen in weeks, and the value of C has always been positive - that is useful information!

Don't email me when there's a profile incomplete warning. Don't email me any time there's an "ERROR" entry, because that just makes people reluctant to use error level logging. Definitely don't email me when there's a unique request complete string, that's trivially different every time. But do let me know when something weird is going on!

I once worked in an envrionment like this that saw about 1k emails a day when I started... What I started doing was trying to resolve one bug a day... I'd look at an error, do a search in my email filtered folder for that error, and manually count them, whichever had the most emails, I took care of it.

About 2/3 just came down to filtering certain classes of errors. 4xx errors, I stopped email notification altogether, since they were already being trapped/handled by the system. Others were a little more specific. Ironically, .Net tends to handle some things that should be 4xx errors as 500, so reclassifying those took out a lot as well.

In the end, within about a month, the emails were down to a manageable 20 or so a day and got more visibility as a result.

Don’t mean to sound snarky, but there are tools that do this and have been for years. If you’ve been grepping through logs for the last 3 years, you’re doing it wrong for the cloud era.

Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it’s simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It’s ultimately an operational problem, not a technical one.

Alas, back in the real world, logging is always the last thing teams have time to think about...

Care to share what types of tools do this? I'm genuinely interested. I haven't come across a log management tool that uses AI to detect abnormal conditions based on the log message contents like the OP describes. I stick to Papertrail for the most part though so I'm likely out of the loop.

I’ve used and really liked DataDog in the past. It has some rudimentary ML functionality for anomaly detection of certain fields, but it’s only getting better.

I’ve also had clients in the past use Splunk with ML forecasting models that inject fields as part of the ingest pipeline. I don’t know the details of that implementation; I just know how the dev teams were using it.

My company uses Sentry.io, which doesn't have AI stuff but does have good tells for separating "normal" errors from "unusual" errors.

There's nothing particularly wrong with e-mailing (or using other push notification mechanisms) when a warning or error occurs. When those logging levels are properly used it means some sort of administrator intervention is needed. But send the notification only once. Don't keep spamming out the same notification over and over.

Honestly, it sounds like there just needs to be some higher discipline here around what the different log levels mean. ERROR should only be used if your system failed to service a request. Any other errors are WARN. Then you tell the system to notify you on any ERROR log, because that means that requests are failing to be processed.

In your example, the `NumberFormatException` is a bad ERROR entry, because it's covered by a WARN entry right below it. Meaning that the request was not failed -- it entered a default value instead of the bad parsing. So that exception should also be at WARN level.

(Arguably, overwriting input values with defaults because of a parsing error is probably a bad idea and should be an ERROR due to rejecting the request. But I'm rolling with what we got here.)

> I still don't know the best way to monitor and alert users for large distributed systems.

I'd imagine that this is somewhere statistical process control would apply: if you're dealing with a system in which errors are expected, then monitor frequency & magnitude, and alert when they fall outside of one or two standard deviations from the mean.

For a financial company, you'd probably want a graph of net worth or somesuch, and alert when it falls outside of one standard deviation. If you don't have the IT do calculate net worth to at least hourly granularity, then get there, and aim for to-the-minute granularity. This shouldn't be hard, but it might be, and if it is then it's worth fixing.

Not only did they leave old code in the system, but they made changes to it without reviewing whether the old code would still work correctly, despite still being callable:

In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called.

If you want to leave the old code in, fine, but then it still needs to be tested.

Reading this I was considering point #6 and shutting down the system. If the proper alerts had shown a system improperly taking on massive positions, and the related risk in dollar terms, wouldn't shutting down be a better route than bankruptcy? In retrospect, would (could?) they have chosen that?

Of course, violating those SLA's could cause bankruptcy through client dissatisfaction but that seems less certain than bleeding out money.

> Its been my experience that regulators care more that you follow the steps in your manual than those steps necessary being the best way to do something.

Not a joke. Bitfinex lost its Bitcoins by being compliant (regulators required that users wallets are displayed on the blockchain). It'd have been safer to just have a single cold wallet.