News for hackers. Stuff that matters.
Re:Binance built a 100PB log service with Quickwit by hanniabu on Thursday July 11, 2024 @11:09AM (#40937570)
I feel like I shouldn't need to mention this, but comparing a news site to a financial exchange with money at stake is not the same. If there is a glitch you need to be able to trace it back and you can't do that with some abstracted metrics.
Re:Binance built a 100PB log service with Quickwit by pixl97 on Thursday July 11, 2024 @12:40PM (#40938381)
Yea, on a news site, the metrics are important. If suddenly you start seeing errors accrue above background noise and it's affecting a number of people you can act on it. If it's affecting one user, you probably don't give a shit.

In finance if someone puts and entry for 1,000,000,000 and it changes to 1,000,000 the SEC, fraud investigators, lawyers, banks, and some number of other FLAs are shining a flashlight up your butt as to what happened.

Re:Binance built a 100PB log service with Quickwit by KaiserPro on Thursday July 11, 2024 @12:50PM (#40938458)
right, and the SEC see that you're mixing verbose k8s logging with financial records, you're going to get a bollocking.
Re:Binance built a 100PB log service with Quickwit by KaiserPro on Thursday July 11, 2024 @12:49PM (#40938449)
You are misreading me.

I'm not saying that you can't log, I'm saying that logging _everything_ on debug in an unstructured way and then hoping to devine a signal from it, is madness. You will need logs, as they eventually tell you what went wrong. But they are very bad at telling you that something is going wrong now.

Its also exceptionally bad at allowing you quickly pinpointing _when_ something changed.

Even in a logging only environment, you get an alert, you look at the graphs, then dive into the logs. The big issue is that those metrics are out of date, hard to derrive and prone to breaking when you make changes.

verbose logging is not a protection in a financial market, because if something goes wrong you'll need to process those logs for consumption by a third party. You'll then have to explain why the format changed three times in the two weeks leading up to that event.

Moreover you will need to seperate the money audit trail from the verbose application logs, ideally at source. as its "high value data" you can't be mixing those stream at all