News for hackers. Stuff that matters.
Re:Ask HN: Best “I brought down production” story? by tasuki on Sunday June 27, 2021 @04:28PM (#27654990)
A client side bug was crashing the server? I always thought the client depended on the server, not vice versa.
Re:Ask HN: Best “I brought down production” story? by Agentlien on Monday June 28, 2021 @03:57AM (#27659154)
It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.
Re:Ask HN: Best “I brought down production” story? by tasuki on Monday June 28, 2021 @09:53AM (#27661979)
> The server lacked validation in this case and crashed when processing the invalid data.

As a backend dev, I'd point to this as the root cause, not the client sending invalid data.

Re:Ask HN: Best “I brought down production” story? by Agentlien on Monday June 28, 2021 @10:03AM (#27662105)
I see where you're coming from. I would say it's the cause of the crash but not the root cause of the issue as a whole. Both obviously need to be fixed.

Fixing only the client side issue removes this crash entirely, but leaves one open for similar issues in the future.

Fixing the server side issue fixes the crash and related ones, but doesn't address the root cause of the faulty data being generated by the client.

Re:Ask HN: Best “I brought down production” story? by mianas on Sunday June 27, 2021 @06:51PM (#27656212)
I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length.

AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.

Re:Ask HN: Best “I brought down production” story? by Agentlien on Monday June 28, 2021 @04:02AM (#27659174)
This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?
Re:Ask HN: Best “I brought down production” story? by mianas on Monday June 28, 2021 @09:45AM (#27661882)
When I read "wasn't entirely correct", and "probability of a crash was about one in a million", I wondered how you'd work that value out.

Predictable statistics screams RNG, to me. Especially if the code author knew it was an imperfect solution, and didn't know of one that wouldn't cause this crash. (I've done stuff like this before.) Also, for the client to crash the server, it'd need to give it some form of bad/confusing data, but for some reason this doesn't happen with fewer users.

I think that was my logic at the time of writing that comment, though I've thought of another few guesses while typing this out/reading your follow-up.