News for hackers. Stuff that matters.
Re:Ask HN: Best “I brought down production” story? by Agentlien on Sunday June 27, 2021 @02:27AM (#27649128)
Here's a good reminder of the dangers of scale.

When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics.

At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it.

After a bit of searching I found the responsible code. It was a client side bug which crashed the server. The line even had a comment mentioning that it wasn't entirely correct, but that the probability of a crash was about one in a million.

Well, we had millions of players.

Re:Ask HN: Best “I brought down production” story? by tasuki on Sunday June 27, 2021 @04:28PM (#27654990)
A client side bug was crashing the server? I always thought the client depended on the server, not vice versa.
Re:Ask HN: Best “I brought down production” story? by Agentlien on Monday June 28, 2021 @03:57AM (#27659154)
It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.
Re:Ask HN: Best “I brought down production” story? by tasuki on Monday June 28, 2021 @09:53AM (#27661979)
> The server lacked validation in this case and crashed when processing the invalid data.

As a backend dev, I'd point to this as the root cause, not the client sending invalid data.

Re:Ask HN: Best “I brought down production” story? by Agentlien on Monday June 28, 2021 @10:03AM (#27662105)
I see where you're coming from. I would say it's the cause of the crash but not the root cause of the issue as a whole. Both obviously need to be fixed.

Fixing only the client side issue removes this crash entirely, but leaves one open for similar issues in the future.

Fixing the server side issue fixes the crash and related ones, but doesn't address the root cause of the faulty data being generated by the client.

Re:Ask HN: Best “I brought down production” story? by mianas on Sunday June 27, 2021 @06:51PM (#27656212)
I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length.

AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.

Re:Ask HN: Best “I brought down production” story? by Agentlien on Monday June 28, 2021 @04:02AM (#27659174)
This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?
Re:Ask HN: Best “I brought down production” story? by mianas on Monday June 28, 2021 @09:45AM (#27661882)
When I read "wasn't entirely correct", and "probability of a crash was about one in a million", I wondered how you'd work that value out.

Predictable statistics screams RNG, to me. Especially if the code author knew it was an imperfect solution, and didn't know of one that wouldn't cause this crash. (I've done stuff like this before.) Also, for the client to crash the server, it'd need to give it some form of bad/confusing data, but for some reason this doesn't happen with fewer users.

I think that was my logic at the time of writing that comment, though I've thought of another few guesses while typing this out/reading your follow-up.