Watch out not to condemn them no matter what they do. Sure, they messed up in the past, but given the situation they found themselves in when they saw this report, this response is still good.

You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.

I disagree. They ignored the issue for years and that part won’t be erased from memory. They should be given credit for finally fixing it, but nothing more. What happened before this week was and remains forever shameful.

Very likely that internally there were people who had an inkling this was going on, but could never convince product to devote time to put it in the backlog. This change is a win for that engineer.

This example will be ammunition in internal debates about how much to weigh product pressure vs engineering suggestions that improvement is technically possible; the next time an eng says "let's look at this slowness one more time, it may be fixable more easily than you think", they might be listened to.

> What happened before this week was and remains forever shameful.

lol without context you'd think something tragic actually happened here.

Losing customers over such a simple, known problem, over years… is tragic for a company like Rockstar. This is like Microsoft ignoring a bug that made Windows take 10 minutes to start and just letting that happen for 5 years. Now tell me that’s not tragic.

Anyway GTA5 is just a game.

To you it's just a game, to Rockstar GTA is literally a multi-billion dollar product with a household name that politicians talk about.

I can't believe you're missing this huge point.

> You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.

On the other hand, a few days of doing the absolute bare minimum of quality control doesn't undo a decade of actively hostile practices. They're still very very deeply in the red when it comes to their relations with the playerbase and for good reason.

Paying $10k to the person who documented this bug is a good start, but you need to do a whole lot of work to make up for the impressions caused by that kind of prolonged systemic negligence.

Right. I'm not saying eval(their net behavior)>0; I'm saying that if your response function is always negative, you lose the ability to influence their response.

Have you heard the one about the country with the death penalty for all crime? It just leads criminals to maximally try not to get caught, without regard to damage done [i.e. removing all witnesses to even the smallest crimes]. This isn't what you want; even within "negative" relationships both sides lose when you don't have a gradient of responses.

The crux of this is that most readers presumably found out about both the bug and the fix in relatively short proximity, and their response is negative to the combined events, not to the knowledge of just the patch.

A criminal prosecuted for murder and petty theft simultaneously can't really go around complaining that they got 25 years in gaol for stealing an apple, but if everyone had already known about the murder and dealt with that accordingly it'd be pretty harsh.

(not that fixing a bug is exactly a criminal act, but the general principle remains)

An alternative possibility: there were 100 bugs of similar consequence, all of which they cared about, and they got 99 of them. This one escaped, because stuff happens.

> I think "shameful" is really more appropriate.

When you make an inadvertent mistake in the future, karma is coming for you: people will speculate that you "didn't give 2 shits".

I’m sorry but no.

This bug existed for a very long time and was the #1 outcry from gamers. It is well documented all over the internet.

Every company I’ve ever worked at and every code base I’ve ever worked on would have made this a #1 priority. If it were a technical limitation of the devs I would agree it is “embarrassing” but this was not outside of their ability.

I simply don’t blame the devs, I blame management. Management should have allocated the resources to fix this truly game breaking bug but they didn’t because money was still flowing. Is it any coincidence that ads played during the 6+ minutes you would wait to get into a game?

Yea, if it was a 6 month old bug or it was a hard to repro edge case, then sure it’s embarrassing. Oops! But years long, and highly visible/reproducible, without correcting? That’s veering into shameful.

Blame should always flow up, but that doesn’t mean there aren’t other causes. For all we know there has been a team of devs putting their kids through college “working” on this for years.

More likely, there were 99 other bugs that they had clearer metrics related to monetization that they prioritized (not that this bug didn't impact monetization - I bet it did, just not in a way they were measuring). Or that their QA engineers identified and flagged as high priority or major new features with their own bugs and regressions or whatever. These things get lost in the shuffle. More likely there were 1000s of bugs in the tracker, probably 10s of thousands.

Yes, I agree with your elaboration. Also, of the 100 bugs, they didn't get 99 of them, because there are surely some left. Maybe they got 90, and there are 9 still hiding out.

The bug was live and wasting paying-customers's time (and apparently spamming them with ads too?) for 6 years.

The game is played by millions of people, we don't need to wonder if there were a lot of more important bugs, this has been by far the most complained about issue for many years.

> This one escaped, because stuff happens.

No, they mostly just don't listen to their players when compared with other game companies.

I understand your frustration, but anyone who's run (or been on) a dev team knows you have a million things you could/should be doing, and many, many priorities. Often supporting new launches/features is the priority.

Sure, ideally there would be someone internal advocating for a fix like this on behalf of customers. But Shameful isn't the right word. It's a miss like any other.

The egg that's on Rockstar's face has more to do with how much money they likely left on the table by not diagnosing and fixing this issue early on. (And assuming they had a team overseeing this aspect of performance with adequate tools to observe and debug it should have been trivial.)

I personally feel like the magnitude of this -- especially compared to how simple the solution is -- makes it a miss that's really much more extreme than most others.

EDIT: Though I agree that "shameful" isn't the right word to use. Rockstar wronged themselves much more so than any other party.

Obviously you are right... but...

If the fix had been some super application specific super obscure thing, we would all understand.

Instead... the bug is easily describable to anyone who knows what JSON is and the fix is “don’t parse JSON the slow way”.

So I think some of the leniency afforded to the devs may be misplaced. Then again, hindsight and all that.

The cause doesn’t actually matter much, usually one doesn’t know the issue until they investigate. So often issues remain open for lack of interest, then someone finally does and it’s a missing comma. This doesn’t affect the fact that the issue has been ignored.

I’m sure GTA has unlimited bugs to look into, but a bug that turns people away before they even start should have probably been given higher priority.

> If the fix had been some super application specific super obscure thing, we would all understand.

Would we?

Even if I don't understand what it's doing, devs doing nothing about an O(n^2) algorithm that takes multiple minutes every load is clearly a big mistake.

Even if it's something that has to be that slow, cache the result.

Oh absolutely, I have a whole host of stuff on my todo list for projects I'm developing that has been there for months and will remain there for months longer.

That said, they're all things like "refactor the frobnicator code so it's more obvious what it does" and "make templating system log changes to server". Nothing on my list says "Look in to that major performance issue that literally thousands of customers complain about every single day". That stuff gets sorted immediately.

If so then they still acted in a gentle way that is rare afaik. If they were so careless they'd just lift the fix and ignore the author.

There probably is a story behind it. Like some developer looked into the bug in the beginning (when when the load time was not that bad because the json file was smaller), saw that the time was spend parsing json, and then, without thinking too much whether this makes sense, wrote in the bug ticket that, to make it faster, they would have to switch from json to something better/faster/different.

Since switching the file format is probably some work, maybe involving many differnt teams and network protocols, the bug was closed because it too much efford.

And then, whenever somebody complained, the old bug was referred which unfortunately never got a proper analysis.

Sad, but I saw this happening many times in the past in some projects. Since the old and wrong analysis will be in many powerpoint slides and heads, it is incredible hard to argue against or get even time to work on such a problem. Usually, you need someone who just takes some time form other projects without telling anyone and ninja fixes the problem to solve it.

Based on my own experiences, I'm absolutely certain many people over many years tried to get the load time fixed, and then were beaten down for their earnestness.

Just as I'm certain that this unsolicited 3rd party fix was used to settle some old scores.

Corporate politics is a blood sport.

you forgot to mention that ads are displayed during the loading screen

I think "embarrassing" could still be the correct word. If you were to ask the development teams responsible they might feel a little embarrassed over the solution being so small (yet technical).

And if they didn't care at all given what you say then why would they care to implement a fix for it all these years later?

This is akin to that guy buying a whole-page ad to complain about his ISP recently, resulting in enough shame to trigger action. It's a bad look for Rockstar considering the income that title generates and at most it was a few weeks worth of dev time. Very interesting to watch the public reaction to this in comparison though, seems to be a lot more sympathy for Rockstar than for ISPs.

They still made like a billion dollars in shark cards so apparently all the people buying shark cards didn't give a shit about the load time issue and that's the real problem.

It's a video game, so maybe give them some slack. We're not talking a SolarWinds breach here.

The game supports 8 platforms, has virtually miles of terrain, and now Rockstar is suppose to feel ashamed because they didn't fix a big performance issue that affected startup times?

The game probably has tens of millions of lines of code.

I'm glad they shipped. If they can enhance the startup time, great, bonus for them. Additional credit for thanking the third party that uncovered the perf issue.

I don't think you've been following this story. If you looked into the technical details [0], and surrounding context, you would understand why your comment isn't particularly insightful. For example, terrain has nothing to do with this bug and it isn't a complex fix that interacts with "millions of lines of code." A single run of a profiler would have caught this.

I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester.

[0] https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...

I think you missed previous news about this. This was an incredibly trivial issue that would show up in any use of a profiler on the slow startup screens. The fact they didn't is that they're either unable to run profilers on their code at Rockstar (which would be shameful indeed) or they never bothered to do so. Either case shows egg on face at Rockstar.

Strange that this was this downvoted. It is an issue that a single dev with a profiler should've been able to identify very easily. That they hadn't addressed it despite it being such an annoyance for so many users for so long is embarrassing.

Not to take away from the feat and initiative of doing so without access to source code.

+1 — it’s an item a single person without access to the source code was able to diagnose and fix.

I’m pretty shocked at all the defending of the game company here, honestly. Highly visible bug, affecting high percentage of users, relatively simple fix. I take pride in the software I wrote and would be personally very embarrassed if I caused this AND didn’t fix it for years. I guess companies don’t have feelings and can’t feel shame but I would personally feel shame as a supposed professional!

There are a lot of excuses for why this wasn’t fixed but no great reasons.

The idea that screwups are "shameful" is poison in any organization. I fight it in the companies I work for, I fight it here.

Bugs happen. Serious bugs happen. They will always happen. Shaming people for bugs undermines institutional robustness against bugs because it incentivizes people to hide mistakes. Furthermore, companies with blame-n-shame culture typically fault users when those users make inevitable "mistakes" statistically guaranteed by bad UX design.

In contrast, transparency is good, acknowledgment is good. Rockstar acknowledged fault and rewarded the reporter, which is difficult. That's a positive institutional arc I'd like to encourage, at Rockstar and in the wider industry.

If you stop at "embarrassing", I'm with you. But when you proceed to "shameful", that's quite different and we part ways.

Just to clarify my point: The bug was a screwup. We’ve all done screwups. My worst bug was an amateur hour royal screwup that resulted in DDOS. Nobody should be shamed for bugs, including serious ones that get to production.

Not fixing a serious, visible, known bug for six years is what’s shameful. There is nothing positive about neglect like that. It’s kind of admirable that they actually rewarded the guy but it shouldn’t have even got to that point.

Totally agree you should not shame for bugs. The shame is for ignoring this bug until one of your customers finally had enough and fixed it by patching the binary. If it was my company I would feel ashamed.

I haven't read all the comments on these articles, but I haven't seen anyone claim that the bug shouldn't have happened, that anyone should lose their job, that no one "good" makes those sort of mistakes, etc. But this was a bug that lasted long enough and irritated enough customers that one of them finally diagnosed and seemingly fixed it without ever looking at their source code. That's one hell of a process failure. And splitting hairs over whether we would feel professionaly "embarrassed" or "ashamed" in a similar situation is pretty silly.

Everybody makes mistakes. Bugs are inevitable.

Managerial problems that allow high-impact, low-fix-time, high-prevalence bugs to exist for 7 years are shameful and are in fact poison to an organization themselves.

Don't mistake the shaming for really being about the bug itself here. The problem isn't the bug, it's the organizational culture that allowed such a massive, easily fixed, widely encountered bug to persist for 7 years in a game that has made them $6 billion dollars over the last few years.

> now Rockstar is suppose to feel ashamed because they didn't fix a big performance issue that affected startup times?

Yes, they should be ashamed of this.

It's true that the game is probably absurdly complex, but how did they never profile loading times?