I wasn’t trying to blame engineers that wasn’t my intention, but someone needs to own establishing the environment that actively encourages or doesn’t encourage curiosity.

Talented people are found everywhere, but hopefully someone with authority over them doesn’t slowly poison the drive out of them.

Why should developers necessarily be trying to cut loading times? Perhaps GTA is phenomenally successful because they have good BAs and POs who are able to identify and convince engineers to work on what matters to the bottom line.

Screw this. Developers should advocate for their craft and for quality, and business metrics should hold an adversarial position to balance things out. We definitely should not be accepting the narrative that developers just do what they are told and don't push back on what we know is good for product - we are in a unique position of power, we should recognise this and use it for the betterment of the industry as a whole. The alternative is we let corporate greed slide our standards of quality into the gutter thereby making the job at hand miserable.

Very much agree with this.

That said given the quality of what they have put out, I'd surmise the issue here is morelikely they don't feel the same pain on load times?

Bandwidth too good, rigs too powerful, dev envs overpowered, etc. to enable developer velocity can often also mask real world problems.

Hell, even hot reload can mean you don't see load as often and are able to easily rationalize as "not that big a deal" - I wouldn't necessarily take it as a sign they are not deeply invested in their craft.

I very much agree with you as well actually.

One of the things that I've instituted at my work is our methodology for acquiring test phones for our product (WhatsApp for hospitals, basically). We go to a supermarket and buy the most eye-catching box of the budget phones available in the phone aisle.

We've squashed so many bugs and performance problems this way. It's cheaper, it instils empathy in product decisions, and our customers love the user experience as a result.

Similarly, one of my first big decisions after joining was to wean ourselves from our AWS addiction using Kubernetes. This also allowed us to design a dev env that fits on a modern machine comfortably, but can scale to entire Hospital Trusts in production. Once again: focusing on portability now means we can sell our product on-prem and hybrid, because we designed our developer UX this way.

I don't know, this seems like it was a pure CPU problem on good hardware. But anyway a developer who has 10 hour days and backlog stretching beyond the horizon will not have idle time to go try things out. It needs to be someone's explicit job to go hunt down things like this.

And given it was data-size dependent, I wouldn't be surprised this was not an issue at the start while someone who's job it was to look for this kind of issue was looking. And the issue appeared after they got retasked to something else.

In fact maybe this is the only kind of issue that you end up with after all is said and done. Just like in WW2 they reinforced the parts of the fuselage on returning fighter craft without bullet holes, because if a bullet hit there, the craft would not return.

> Developers should advocate for their craft and for quality, and business metrics should hold an adversarial position to balance things out.

What makes you think this isn't the case? Business metrics holding an adversarial position over other interests obviously doesn't preclude business metrics defeating other interests.

In my experience, business interests always override everything else. Often times to the detriment of the business, and often as soon as mere 3 to 6 months in the future.

Sounds like an extremely false dichotomy. There is also the solution that you set up your own company, or convince your manager your time is best spent on this.

I’ve worked around too many startups ruled by developers. They’re being run into the ground because those developers think technical NFRs as defined by them are the be all and end all. They’re not. Those same developers will happily spend months fiddling with the system without making any actual improvements and call it “paying down technical debt”.

Give me a PM (or good engineer) that understands the bigger picture every time.

> There is also the solution that you set up your own company, or convince your manager your time is best spent on this.

This is the definition of a flippant comment. Why should society require we become the lord of our own domain or a political chameleon just so that we can advocate for decent standards? Should people who live on flood plains "just move" when the tide comes in? Ridiculous.

Understanding the bigger picture includes understanding the developer's point of view.

> I’ve worked around too many startups ruled by developers. They’re being run into the ground because those developers think technical NFRs as defined by them are the be all and end all. They’re not. Those same developers will happily spend months fiddling with the system without making any actual improvements and call it “paying down technical debt”.

Sounds like those super-clever managers didn't understand what the hell was going on to me. Perhaps they should've held their nose and tried to learn about their product so they can understand and communicate that bigger picture they speak of instead of abdicating their adversarial role and leaving dev teams rudderless.

What makes you think they didn’t understand the developers’ point of view? Maybe they understood it, but didn’t agree with it.

> Give me a PM (or good engineer) that understands the bigger picture every time.

I was referring to this sentence: you're now telling us that the ideal good engineer understands the developer's point of view but doesn't agree with it. It sounds like you just want to be surrounded by yes men to me. The perfect recipe for a failed startup.

Maybe, just maybe, the developers are sometimes wrong about where they should spend their time.

Even if your parent comment presented false dichotomy, you responding with the other extreme isn't convincing.

As a senior programmer I respect business priorities and I don't go fixing technical debt for months.

That being said, we the people suck at moderation. See, the problem I faced was exactly the opposite of what you encountered: I've been too accommodating and allowed myself to be micro-managed to the point of a manager telling me how should I do my job; that's when it finally clicked with me on what a toxic workplace have I landed back then.

So neither of both extremes is good. I agree business priorities have to be respected; absolutely. At the same time we shouldn't allow the businessmen to tell us we can't ever fix technical debt.

Exactly. Managers are there to facilitate people and communication, business leaders are there to facilitate capital and strategic vision, engineers actually make the thing and master the detail in the process. Good businesses learn to listen: not just to their customers, but also to their domain experts.

This is why I try to hold developers accountable for actions their software performs or allows, like auto-banning accounts or not supporting billing limits.

Loading times were the #1 complaint from users about the game. I personally wouldn't play a game where 5 minute loading times were the norm, 10+ minute times are common, and 15+ minute times are not unheard of.

It's the #1 complaint. The thing that makes this egregious isn't so much that they didn't fix the #1 complain, the thing that makes it egregious is that it's a simple enough flaw that a mediocre developer could have just run WPR/WPA against it and see that 90% of those 5+ minutes was spent in strlen. Maybe it's not their area, maybe they don't know how to fix it, but they will recognize that it's a problem with a likely easy fix.

So even if we completely ignore the fact that it's the #1 complaint, it's still egregious that it's been allowed to go on for this long.

You've clearly never played it nor read any of the community discussions about the game. The #1 complaint is (or was) loading times. It's why I've personally stopped playing years ago.

I play GTA, am annoyed by those loading screens, and have found those forums while looking for solutions.

Companies are successful not because they delight every customer consistently, but because they do just enough to make the customer spend money with them.

After all, I will still buy the next version.

I mean that's fair, but I'm talking about wanting to be curious and encouraging environments where curious individuals can be lauded instead of dismissed. Not everyone on a team HAS to be curious but for those that aren't they shouldn't be smacked upside the head with corporate bureaucracy.

You are assuming curiosity was the problem instead of a lot of other possible factors like bandwidth.

I wasn't blaming this problem solely existing due to being incurious but more of lamenting how corporate environments don't encourage people to be curious.

That may have been your intention, but your statement of "environments where people aren't curious" comes across as putting the blame on the people themselves. As has been stated elsewhere, often times engineering is lacking in bandwidth to do anything but meet the deadlines.

I don't disagree that the "environment" itself could do more to further making improvements outside of the assigned work - initiatives such as 20% time, cafe days, etc. exist in other companies. But I don't put the blame on the curiosity of the people themselves.

I mean at the end of the day we’re products of our environments.

If SWEs can only work on features and do nothing else that is still a sign of poor project management. Just accepting that fate is worse for your growth I’d wager.

But idk, I never worked in an environment that encouraged curious people either. I suppose the closest for me was working at Comcast but that was only one manager with a small team of 4 people and no real deadlines.

I feel like two separate thoughts came from replies to my comment. One advocating for engineers to push back and care about their crafts, the other absolving the choices that were clearly made by people as some mechanized process that could never be changed.

The only thing I’m wondering is what the replies would be like 10 years ago on HN.

Game was out for almost EIGHT YEARS by the time this bug was fixed. There can be no reasonable explanation for them not fixing that. 5 minute load every time you try to go into multiplayer. The amount of revenue lost to that one single bug is staggering. This bug was a reason why I never played GTA online, for example.

Performance issues are one of those things that are easily waved off with broad explanations like "well there's a lot of assets to download", "their internet connection may be slow" etc. it may not have been clear to anyone at that stage that it was a "bug".

The underlying organizational problem then becomes the fact that performance isn't priority number 1. I would suspect that if the perf was taken seriously, some time spent on profiling would immediately reveal this issue.

Performance is not priority number 1. Number 1 is to ship the game on schedule so you can stop crunching for the first time in months, number 2 is to make an interesting game that people pay for and justifies your not getting laid off. The games industry is not a particularly safe place to be in from what I understand.

It is (probably) taken seriously, I am sure they were pushing things to limit to render just one more car or have a one more AI etc. My assumption the loading times just somehow slipped off and no one thought it was a solvable problem or it is just loading performance is not taken as seriously

> My assumption the loading times just somehow slipped off and no one thought it was a solvable problem or it is just loading performance is not taken as seriously

That does sounds like awfully like confirming that indeed no-one was curious, because they were too busy chasing KPIs or whatever other metrics management were using.

If was the number one complaint about the game. R* is one of the biggest studios around. It would have cost them nothing to just hire a developer whose only task is to hound down that bug and fix it.

It could've been waved off if there wasn't single player part of that same game that loaded few times faster.

That no one on the team maintaining the game looked into it likely means they don’t play the live game themselves. Developers and managers on up the chain.

Understandable given the heavy crunch it sounds like Rock Star works in. If you’re a developer spending 60+ hours a week working on it, you’re not going to play that in the few personal hours left over.

Your argument is a valid reason why this bug wasn't fixed when the game was first released. But no crunch lasts 8 years. And complaint about loading times were everywhere for all this time.

There are people that are paid to play the game (testers) or collect feedback from the people that plays it. Hard to imagine that they were oblivious to the problem

>The amount of revenue lost to that one single bug is staggering.

The players who are still there are used to it so the value is likely near zero. It's the sort of thing that's 100% worth fixing before launch or soon after so you don't lose players in the first place. But the fix loses value every day it goes unfixed because you're already lost those load time-sensitive players. Keep in mind that years after release it still makes Rockstar/Take-Two millions per day (https://www.thegamer.com/gta-5-rockstar-2-5-million-per-day/) so I guess a lot of players just don't care.

For the record: I'm surprised anyone is willing to tolerate that shit.

Lots of people cite long load times as why they stopped playing GTA5. They are certainly making bank, but they also certainly left money on the table because those players who left could have still been there and spending money.

Them making a lot of money doesn't prove that they couldn't have made more money. Everyone knows better performance means better conversion which means more $$$.

It is definitely non-zero, I think since the launch of game the number of revenue lost to this single bug is in tens, possibly hundreds of millions of dollars. Considering that game made somewhere around 10 billions USD, 1-2% of total revenue lost to loading time seems plausible, given how horrible the problem was.

GTA Online player count statistics indicate 2.5 million GTA Online players every month. A reduction in total waiting time for all the players of 18.5 years every month.

That's assuming each player only logs once in a month, which is definitely not the case.

I was trying to make the fact nobody decided to fix it for years, less outrageous that it is...

I agree with the sentiment. However, it is a tad surprising when you consider the productivity hit, even just internally speaking, that ~5min of cpu time in strlen() on every start of the app incurs.

How much time did perfectly qualified engineers, who could have attached a profiler at any moment, spend sitting at a loading screen spending most of its time in unnecessary strlen() calls? I understand we see this through 20/20 hindsight, serendipity can work against you, etc., but that's a crazy irony.

A toxic culture resistant to change can obtain these results no matter how brilliant/skilled the people.

It's true, but also, you can get this result with a competent team and good culture, curious people, etc. You could just have bad luck and nobody happened to check it.

You might say that assuming this was due to inherent incuriousity amongst the developers is quite incurious.

it was a feature - they served ads during loading screens