News for hackers. Stuff that matters.
Re:Read Every Single Error by abraae on Wednesday May 17, 2023 @10:05PM (#35983436)
I guess my age is showing but why would you not read every error? Under what circumstances is it acceptable to ever use a metric like raw number of e.g 500 errors, if you don't know the breakdown of them, whether you're dealing with 50 different errors vs the same error happening 50 times?
Re:Read Every Single Error by toast0 on Wednesday May 17, 2023 @10:13PM (#35983470)
It kind of depends on the types of errors. Do you need to do anything about every connection reset by peer error for web clients? Could you even if you tried?

OTOH, the correct number of connection reset by peer errors between your web servers and your database servers is likely zero, and there may be something to do.

Re:Read Every Single Error by abraae on Wednesday May 17, 2023 @10:19PM (#35983497)
There are different types of errors as you rightfully point out. Peer resets for web clients are (typically) caused by environmental factors outside the hosts control. It would make sense to group then and monitor then in the aggregate - if there's a surge of 200% then some investigation might well be warranted.

I will however guarantee that the author does not mean that they actually look at every peer reset individually.

Broken database connections - yes, a code red and they must be examined urgently.

The whole point is to classify your errors, and especially to know when a new class of error had been seen.

Re:Read Every Single Error by eru on Wednesday May 17, 2023 @10:08PM (#35983449)
At Google, we couldn't have read every single error that eg our Google Photos servers produce, even if we wanted to.

At that scale, you get some errors even purely from bits flipping randomly in memory due to cosmic rays. (The latter is speculation. I don't remember how many of those we got, or whether we used enough error correcting memory. It's been a while.)

Re:Read Every Single Error by mteigers on Wednesday May 17, 2023 @10:24PM (#35983519)
Ditto, my managed service at Google threw close to 3k 500s / second when I was still there. Anything from cosmic rays, faulty hardware bit flipping, hard drive failures.

We did, however, aggregate and group similar 500s and those did get looked at, but no way could we have looked at all errors.

The other thing, is that with resilient infrastructure, who cares about an occasional 500. Back off and retry. No harm done.

Re:Read Every Single Error by willcipriano on Wednesday May 17, 2023 @10:27PM (#35983536)
3k? What percentage of requests is that?
Re:Read Every Single Error by samus on Wednesday May 17, 2023 @11:27PM (#35983849)
User experience might indeed not be influenced by these errors, but errors of a less stochastic nature will impact it. The former obscure visibility of the latter, and that's probably the point of TA.
Re:Read Every Single Error by MichaelZuo on Thursday May 18, 2023 @05:54AM (#35985936)
It's likely some fraction of those errors would have had some tangible impact on user experience.
Re:Read Every Single Error by abraae on Wednesday May 17, 2023 @10:20PM (#35983501)
Surely you would have wanted to know how many bit flip errors you were getting - surely you attempted to isolate them, count them?
Re:Read Every Single Error by eru on Wednesday May 17, 2023 @10:58PM (#35983692)
Sure, you do statistics on them and you look at random samples in detail. But that's different from 'Reading Every Single Error' like the submitted article suggests.
Re:Read Every Single Error by cortesoft on Wednesday May 17, 2023 @11:13PM (#35983767)
Counting and grouping them is the opposite of what the article is arguing for. The whole point of the article is that you shouldn't be doing any aggregation and should be investigating each one individually.
Re:Read Every Single Error by samus on Wednesday May 17, 2023 @11:35PM (#35983909)
But TA also argues that people reach for these aggregation tools way too early. I'd say the author is honest enough to refrain from statement about Google scale because they haven't experienced it. There is value in keeping error rates down even at that level.
Re:Read Every Single Error by cortesoft on Wednesday May 17, 2023 @11:46PM (#35983953)
Sure, but the comment I was responding to was missing the point, because the article wasn't trying to convince people that ignore errors to record and aggregate them, they are arguing that people who record and aggregate errors should instead look at each one individually.

Pointing out that errors should be counted is tangential to the argument at hand.

Re:Read Every Single Error by bobbylarrybobby on Wednesday May 17, 2023 @11:08PM (#35983735)
Curious whether there were any attempts to “read” them with AI. Is this something a LLM could do these days?
Re:Read Every Single Error by samus on Wednesday May 17, 2023 @11:15PM (#35983779)
The article admits that it might not be feasible at Google scale, and that it makes sense to aggregate errors at that point. But surely that works better the fewer uninvestigated classes of errors there are, no?
Re:Read Every Single Error by cortesoft on Wednesday May 17, 2023 @11:12PM (#35983755)
I work for a CDN, we are handling billions of requests a day. With 10s of thousands of servers around the world, we are always going to be having thousands of errors for all sorts of reasons outside our control. No way we can look at all those individually.
Re:Read Every Single Error by arp242 on Wednesday May 17, 2023 @11:31PM (#35983885)
These are probably different kind of "errors" than what the author of this article is talking about. The kind of errors you'll see are dropped network connections, malformed data, and stuff like that which is basically impossible to prevent because it's an imperfect universe.

Even at a fairly small scale you will see "errors" of this kind regardless – you don't need to be a huge CDN for that.

But what the author is talking about – in my reading anyway – is programming errors: exceptions, out of bounds access, explicit report_error() calls, things like that which will result in 5xx errors for the client. You've got a lot more control over that. Sometimes random failures beyond your control will cause a 5xx error, and IMHO that's a bug and this usually should be a 4xx or something else (e.g. if the client sends wrong or junk data you shouldn't respond with a 5xx).

I suppose that's the corollary to this article: not all "errors" are the same, and you should distinguish between different types of errors.