For what it's worth, 10MB of JSON is not much. Duplicating the example entry from the article 63000 times (replacing `key` by a uuid4 for unicity) yields 11.5MB JSON.
Deserialising that JSON then inserting each entry in a dict (indexed by key) takes 450ms in Python.
But as Bruce Dawson oft notes, quadratic behaviour is the sweet spot because it's "fast enough to go into production, and slow enough to fall over once it gets there". Here odds are there were only dozens or hundreds of items during dev so nobody noticed it would become slow as balls beyond a few thousand items.
Plus load times are usually the one thing you start ignoring early on, just start the session, go take a coffee or a piss, and by the time you're back it's loaded. Especially after QA has notified of slow load times half a dozen times, the devs (with fast machines and possibly smaller development dataset) go "works fine", and QA just gives up.
In GTA V, when I tried to enjoy multiplayer with my friends the abysmal load times were what killed it for me.
You actually have to load into the game world - which takes forever - before having a friend invite you to their multiplayer world - which takes forever, again.
So both a coffee, and a piss. Maybe they fixed that now?
This could easily compete for the most expensive bug in history, up there with the Pentium Bug. It might have halved the revenue potential of a billion dollar franchise.
It kind of baffles me that they haven't bothered to fix this trivial issue when the result is to cut 4 entire minutes of loading time.
I never learned the "other side" of this story, but a few years later the same dev team tried to recruit me at a CS contest, to which I politely declined.
More details: I was young, without credit card, and gaming on a mac. AA was free and mac compatible. For a while -- apparently mac ports of unreal engine games were approximately all done by a single very productive contractor and from what I understand the US Army, uhh, stopped paying him at some point. So he stopped releasing the mac ports. From my point of view, this meant that I could only play with other mac users and couldn't use any of the fancy new maps. Logs indicated that the compatibility problems with the new maps were not particularly deep, so I got to parsing the unreal map files and was able to restore compatibility by deleting the offending objects. I implemented texture decoding/encoding mostly for curiosity and because textures were well documented in the reverse engineering "literature." I imagined a workflow where someone would bulk export and re-import textures and aside from the texture names the one piece of metadata I needed was the format: RGBA (uncompressed) or DXT (compressed)? I realized that I could easily identify DXT compression from the image histogram, so I didn't need to store separate metadata. Nifty! But it didn't work. Lots of textures stored in uncompressed RGBA8888 "erroneously" round-tripped to DXT. After poring over my own code, I eventually realized that this was because on many of the textures someone had enabled DXT compression and then disabled it, dropping the texture quality to that of DXT while bloating the texture size to that of RGBA8888 (other textures were still stored as DXT, so compression itself was still working). I wrote a quick tool to add up the wasted space from storing DXT compressed textures in uncompressed RGB format and it came out to about half the total disk space, both before and after the top level installer's lossless compression. They could have re-enabled compression on most of the textures where they had disabled it without loss in quality, and if they had wanted a list of such textures I would have been able to provide it, but it didn't go down that way. When I shared what happened with my father, who had served, his reaction was "Now that's the Army I know!"
Reminds me on loading "G.I. Joe" (from Epyx) on the C64 with a 1541 floppy disk. However, the long loads came after every time you died and meant you also had to swap 4 disks.
To be fair to GTA V, I don't think my installation was on a SSD because it was 50GB or something at the time (now it's 95GB?), but that said when it released SSDs were not as cheap or widespread as they are now so that's their problem. The linked article shows the initial load to be much shorter which did not match my experience.
Many 8bit Atari owners will have horror memories of aborted Tape loads. After over 20 years someone finally discovered a bug in original ATARI Tape loading ROM routine resulting in randomly corrupted loads, no amount of sitting motionless while the game loads would help in that case :)
Which meant the game displayed "rewind to mark 500 and then hit play" and you had to do that to restart lolsob.
I suppose devs dont care about that as long as their QA allows it.
Call of Duty: Black Ops Cold War is around 200GB fully installed with all features (HD textures etc).
Some people have insinuated this is intentional to crowd out other games on console hard disks and make moving away from CoD have an opportunity cost. It's probably just laziness.
I haven't looked into it in the past but some prior offenders had a lot of space wasted from uncompressed multi-lingual audio. Instead of letting us choose a language it installs them all so you can switch in game, and uncompressed for saving the CPU for game logic. For CoD the optional HD texture pack is 38GB so that's still a lot unaccounted for.
It's not like decoding audio takes enough time on any modern multi-core processor to disrupt the game loop. It's not even on the radar.
Is that... the same problem? Is microtransaction data different in your friend's multiplayer world than it is in the normal online world?
Had been like that for nearly a year. A few minutes of work brought out client js file from 12MB to less then 1mb.
Most often I use unicode strings in unexpected alphabets (i.e. from languages that are not supported by our application and that are not used by the mother tongue of any developer from our team). This includes Chinese, Malayalam, Arabic and a few more. There was a time when I wanted to test the "wrong data" cases for some deserialising function, and I was part annoyed and part amusingly surprised to discover that doing Integer.parseInt("٤٣٠٤٦٧٢١") in Java does parse the arabic digits correctly even without specifying any locale.
Is that a library? Or the string "Chuck Norris"?
It certainly caught my attention.
I also had a customer discover a playboy center page at the end of a test we sent to them. One of the dev thought it'd be a nice reward. Things went very bad from him right after the phone call...
Had someone at work mention they needed to drop off a dog at grooming on the way to pick up Chinese.
I had to walk away for a bit, as I couldn’t hold it in, but didn’t know if that sort of humor would fly with that crowd.
I'm sure there are libraries to parse json in C++ or at least they should have built something internally if it's critical, instead they have someone less experienced build it and not stress test it?
There certainly are, but adding a library is much more difficult in C++ than pretty much any other language which seems to tempt people into hacky self-built parsing when they really ought to know better.
https://github.com/nlohmann/json
I know because I used it years ago in school.
One of the fastest libraries (https://github.com/miloyip/nativejson-benchmark#parsing-time) is also header-only and compares its speed to strlen.
Might be, but this particular issue has been raised by thousands of players and ignored for *years*.
The only possible explanation is that management never made it a priority.
I could see this happening. A project/product manager thinking "We could spend $unknown hours looking for potential speedups, or we could spend $known hours implementing new features directly tied to revenue"
Which is kind of ironic since this fix would keep players playing for more time, increasing the chances that they spend more money.
I think we need a new vocabulary to cover situations like this one. It's not just that other issues took priority here, it's that this wasn't even entered into the list of things to give a crap about. It's something like digital squalor.
I really liked this article, but I am a bit surprised that this made it into production. I have seen a few instances of this type of slowdowns live for very long, but they tend to be in compile times or development workflow, not in the product itself.
Boiling the frog, as it were. This class of problems is why I want way more charts on the projects I work on, especially after we hit production. I may not notice an extra 500ms a week, but I'm for damn sure going to notice the slope of a line on a 6 month chart.
I mean I can understand it, a lot of extensions don't need to be on the critical path.
But at the same time, I feel like Chrome could do things a lot better with extensions, such as better review policy and compiling them to wasm from the extensions store.
confirmed by Tampermonkey dev.
The only thing most game companies do when it comes to external libraries is to copy the source code of it into their repo and never update it, ever.
OpenSSL is this way, it's a required installation for Playstation but debugging it is seriously hard, and perforce (the games industries version control of choice) can't handle external dependencies. Not to mention visual studio (the game industries IDE of choice..) can't handle debugging external libraries well either.
So, most game studios throw up the hands, say "fuck it" and practice a heavy amount of NIH.
[0] Despite it being an obvious anti-pattern that you aren't going to update dependencies that require copy/paste and manual merge reviews, so security problems should be obviously more rampant than in systems where updating a dependency to the latest security patch is a single install command line (or update button in a GUI), there still seems to be so many C++ devs that love to chime in to every HN thread on a package manager vulnerability that they don't have those vulnerabilities. They don't "have" dependencies to manage, no matter how many stale 0-Days they copied and pasted from outside projects, they don't count as "dependencies" because they are hidden who knows where in the source tree.
The best algorithm for small, medium or a large size are not the same and generally behave poorly in the other cases. And what is small? Medium? Large?
The truth is that there is no one size fits all and assumptions need to be reviewed periodically and adapted accordingly. And they never are... Ask a DBA.
Yes. That is literally the entirety of the issue: online loading takes 5mn because there are two accidentally quadratic loops which spin their wheel.
> The best algorithm for small, medium or a large size are not the same and generally behave poorly in the other cases.
“Behaves poorly” tends to have very different consequences: an algorithm for large sizes tends to have significant set up and thus constant overhead for small sizes. This is easy to notice and remediate.
A naive quadratic algorithm will blow up your production unless you dev with production data, and possibly even then (if production data keeps growing long after the initial development).
The problem is that when you double the amount of stuff in the JSON document, you quadruple (or more) the scanning penalty in both the string and the list.
Why quadruple? Because you end up scanning a list which is twice as long. You have to scan that list twice as many times. 2x2 = 4. The larger list no longer fits in the fast (cache) memory, among other issues. The cache issue alone can add another 10x (or more!) penalty.
Well, that is an abuse of the term, by people that sometimes don't actually know what that really means. Up to a point, quadratic IS faster than linear after all for example. Too many developer love too abuse the word blindly.
If it is badly tested with no data, it is badly tested with no data. Period. Not "quadratic".
> The problem is that when you double the amount of stuff in the JSON document, you quadruple (or more) the scanning penalty in both the string and the list.
My point was precisely it depends on the data and initial assumption are to be routinely revised. I was making a general point.
Maybe the guy was pinky-sworn that the JSON would hardly change and that the items were supposed to be ordered, sequential and no more than 101. For all you know it is even documented and nobody cared/remembered/checked when updating the JSON. But we don't know, obfuscated code don't comes with comments and context ...
Or, it is actually a real rookie mistake. It probably was, but we don't have all the facts.
There is absolutely no guarantee that a quadratic algorithm has to be faster than a linear algorithm for small N. It can be, in some situations for some algorithms, but the complexity class of the algorithm has nothing to do with that. A quadratic algorithm may well be slower than a linear algorithm for any N.
The only thing the complexity class tells us is that starting from some N the linear algorithm is faster than the quadratic algorithm. That N could be 0, it could be 100, or it could be 1 billion.
In my experience it's usually between 0~1000, but again, that depends. The complexity class makes no such guarantees. The complexity class tells us the general shape of the performance graph, but not exactly where the graphs will intersect: this depends on the constants.
> If it is badly tested with no data, it is badly tested with no data. Period. Not "quadratic".
It is both. The problem is that the algorithm has quadratic complexity. The fact that it was badly tested caused this fact to remain hidden while writing the code, and turn into a real problem later on.
The problem with this argument is that if the data size and constants are sufficiently small enough people don't care about whether the linear algorithm is slow. In the case of JSON parsing the constants are exactly the same no matter what string length algorithm you use. Thus when n is small you don't care since the overall loading time is short anyway. When n is big you benefit from faster loading times.
I honestly don't understand what goal you are trying to accomplish. By your logic it is more important to keep short loading times short and long loading times long rather than do the conventional engineering wisdom of lowering the average or median loading time which will sometimes decrease the duration of long loading screens at the expensive of increasing the duration of short loading screens.
I disagree. If there are no internal incentives for the people who know how to fix this to fix it, or if there's no path from them thinking fixing it could improve revenues to being assigned the ticket, things like this won't get fixed. I can fully believe the load times will result in fewer users and lower expenditure.
I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.
In games the added reason to be slow is that game code is by definition some of the least mission critical code one could find (competes with 90% of the internet web code). Your Linux or Windows code might run a hospital's infrastructure or a rover on another planet. A game on the other hand can launch with bugs the size of your windshield, and can stay like that forever as long as people still pay. And people will pay because games are not unlike a drug for many people.
As such most game coding teams and coders are "trained" to cut every corner and skimp on every precaution. They're not needed beyond a very low baseline as far as software is concerned.
Look at the amount of bugs or cheats incredibly popular games like GTA or CoD have. These are billion dollar a year franchises that leave all this crap on the table despite all the money they make. They have all the resources needed, it's a conscious call to proceed like this, to hire teams that will never be qualified enough to deliver a high quality product and will be encouraged to cut corners on top of that.
Source: a long time ago I worked for a major game developer in a senior management role (unrelated to the dev activity) and left after feeling like "facepalm" for too long in every single SM meeting.
https://randomascii.wordpress.com/2021/02/16/arranging-invis...
I've seen plenty of interesting bugs, best I found personally was a compiler that was outputting files 1 byte at a time.
Games are riff with these sorts of bugs, but the volume of released games vs other types of software makes any sort of comparison unfair.
No comparison is entirely fair but I think your objection is unfounded. The quantity of games being released is irrelevant when we're talking about the quality of their code. Which is really bad for most big titles.
If anything the comparison was unfair in the other direction. Sure, an OS or browser have a lot of bugs. But if games and their code had a fraction of the scrutiny something like Windows gets you might just find out that 4 lines of code were written "by the book". It's something any honest game dev will confirm, game code is a stack of spaghetti code on top of more spaghetti code. The philosophy is that you can just go ahead with bad code because you can always fix it with a patch later on. Then you notice there's no widespread pushback from gamers (because unless it's an absolute bomb there won't be) and move on with the next round of "features", nobody has time for fixing bugs or combing the spaghetti.
One other problem is that eventually some people coming from the gaming industry will end up switching to other types of software development but will stick to the philosophy. I've done a lot of hiring over my career and of one thing I'm certain. Whenever someone came with most of their career in game development or most of the recent experience I asked for the CV to be put at the bottom of the stack. It was a lesson I learned the hard way.
Issues like the low criticality of game code, the "crunch" work style, the idea that you should just get it out there as quickly as possible, the lack of serious scrutiny into the matter, etc. all compound each other to create a coding (and coordination) style that's hard to shake off.
I should have clarified.
Doing an absolute numerical comparison in terms if "bad game code" to "bad B2C code" is unfair, because now days there is a lot more game dev code than B2C code (if we ignore the web, which is a rather lot of code to ignore I'll admit :) ).
All that said, most of the games on my phone run very well. Most of the games on my PC run very well. Do the massive over budget AAA titles have issues? Of course. Human organizations are really bad at creating large projects, we all know that social interaction overhead scales non-linearly with # of people, large projects get bogged down.
(Now throw in B2B software where the CTO makes the purchasing decisions and only people lower level in the company have to use it. All of a sudden enterprise software starts to suck as much as AAA games!)
> The philosophy is that you can just go ahead with bad code because you can always fix it with a patch later on.
Sure, for AAA games. But plenty of games that I play don't have those sorts of issues.
Meanwhile my Windows start menu only opens every other time I press the start menu button. This happened for years, then it was fixed for a bit, then a few patches later it started happening again.
The start menu on Windows should be the 2nd to last thing that breaks, right behind the mouse driver.
Without huge testing efforts, large software projects will collapse under its own weight. The type of software isn't material to the problem.
> One other problem is that eventually some people coming from the gaming industry will end up switching to other types of software development but will stick to the philosophy.
I've hired plenty of ex-game devs, I agree they have some holes in their skills. But I've also found out that if you sit them down they can write code that traditional software engineers couldn't manage. Spending too long in any one paradigm fixes ones mindset into that paradigm.
I also agree that the quality of code someone puts out is highly influenced by the environment they work in. But I've worked with plenty of ex game devs who take pride in writing correct code the first time around.
> Whenever someone came with most of their career in game development or most of the recent experience I asked for the CV to be put at the bottom of the stack.
That's a shame, because I wouldn't have accomplished some of the things I've done in my career if I hadn't hired a few game developers along the way! I've seen some stupid crazy stuff get pulled off, code that "shouldn't be able to exist", but did exist, and was rock solid reliable, all written by game developers.
People are people, and sure 9 out of 10 the person who bangs on kernels for a living is probably super careful about every line of code they write, but I'm not going to judge an entire field because the most bureaucratic outputs of that field are horrible.
That'd be like saying everyone who writes web apps is an incompetent developer because Jira is slow.
Or that all software developers are bad because, honestly, the majority of users only notice what we make when it breaks.
Well, we only hear about the small % of games that have huge problems, not the larger number of games that work just fine out of the box.
Any sufficiently large institution, over time, will prioritise self-preservation over achieving their core mission. This is sociology 101. Once a company has enough users to make it hard or impossible to measure immediate performance, self-preservation is achieved with internal manoeuvering and selling to execs.
This doesn't happen because it would reduce the power of top decision makers and potentially impact profits. e.g. a customer might ask for a chronologically ordered timeline on Facebook, but that would harsh engagement metrics, revenue, etc. If stuff like this did happen more often though, you'd get products and services that more often achieve their stated aims.
Normally the response from management was "but will that increase our profits?", to which my answer was "eventually, yes"
After tangentially working on this for a long time I'd say that the core issue is so deeply ingrained in human psyche that it may not even be a matter of education (starts early), let alone organization (starts happening when everything else is "set in stone"). There's no organizational structure that fits the types of activities we humans tend to do these days and that can deliver low latency, consistent results at scale. We mitigate issues in one area by accentuating them in others.
You can have one large flat structure but the load on the single coordinating circuit (the manager) will compromise quality. You can split the thing in multiple, individually coordinated units but the added layer of coordination and latency will compromise performance.
Maybe some form of general purpose but not quite full AI, something that combines human like intelligence and engineering like consistency, might be able to do what humans are supposed to but without the variability of humans (which is both good and bad).
We don't really know what it is simply because we haven't introduced any "real" AI anywhere, let alone:
>> some form of general purpose but not quite full AI
Talking about efficiency as you scale up large organizations, it's inevitable that humans will introduce delays and variability in the work which cannot be eliminated because it's human nature, biologically and psychologically. Since humans can't change on a timescale that makes this discussion relevant, the only way for very large organizations (like large companies or governments) to operate just as efficiently as small ones is if they rely (quasi)exclusively on some AI that's as capable as top individuals at delivering results but with fewer of the drawbacks. Not only would it not operate as 100.000 distinct entities but as one or a few, it would also run consistently and predictably.
The HN discussion[1] was started by an article that provided numbers that seemed to suggest that Wikipedia's spending was slowly spiraling out of control.
What really stuck out is that Buffet said its the most important thing about Business and he didn't learn it in Business school. Same! I did an MBA and it was the biggest elephant in the room.
"if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right"
Like anything else, things will be as bad as the market allows. So I'd expect monopolies to do a worse and worse job of making good decisions and companies in competitive fields to do a better and better job over time. Thus the difference between TakeTwo and Facebook, and the need for lower cost of entry and greater competition in all economic endaevors where efficiency or good decision making is important.
The messenger website has been atrocious for me lately. On my high-powered desktop, it often lags a bit, and on my fairly high-end laptop, it's virtually unusable. I thought it must be something I changed in my setup, but it's oddly comforting to hear that I'm not the only one with such issues.
It's funny. That page says "A simple app that lets you text, video chat and stay close to people you care about." If it's so simple, then why does the browser version of the site take forever to load?
If it ever reached the point where it had to be an item in a priority list, it's already a failure. Some developer should have seen the quadratic behavior and fixed it. It's not the type of thing that should ever even be part of a prioritized backlog. It's a showstopper bug and it's visible for every developer.
Is GTA online still attracts new users in droves? I doubt.
If the old users live with the loading time for years, they are likely to continue living with it. It would be nice if Rockstar fixes it, but I doubt it would be anything except a PR win.
2. we can be reasonably sure people will buy newer GTA installments regardless of whether this bug is fixed or not.
but:
3. if there's still money to be made from microtransactions this is a huge issue and would absolutely be worthwhile, imo.
In fact, I think the iOS app for FB Messenger did get a redesign due to problems and it’s rewritten from scratch? I remember being pleasantly surprised after the big update… It became lightweight, integrates well with iOS and supports platform features.
On the other hand, the desktop app or the website is a shitshow :-(
Management: So, what can we do about the loading times?
Programmer(s): That's just how long it takes to load JSON. After all, the algorithm/function couldn't be more straightforward. Most of the complaints are probably coming from older hardware. And with new PC's and next-gen consoles it probably won't be noticeable at all.
Management: OK, guess that's that then. Sucks but nothing we can do.
Management had no idea of knowing whether this is true or not -- they have to trust what their devs tell them. And every time over the years someone asked "hey why is loading so slow?" they get told "yeah they looked into it when it was built, turns out there was no way to speed it up, so not worth looking into again."
And I'm guessing that while Rockstar's best devs are put on the really complex in-game performance stuff... their least experienced ones are put on stuff like... loading a game's JSON config from servers.
I've seen it personally in the past where the supposedly "easy" dev tasks are given to a separate team entirely, accountable to management directly, instead of accountable to the highly capable tech lead in charge of all the rest. I've got to assume that was basically the root cause here.
But I agree, this is incredibly embarrassing and unforgiveable. Whatever chain of accountability allowed this to happen... goddamn there's got to be one hell of an internal postmortem on this one.
Programmer: Loading times are really slow, I want to look into it next sprint.
Management: Feature X is higher priority, put it in the backlog and we'll get to it.
Aside from being generally shameful, the real kicker was that this was a "website performance & reliability" company x__x
Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN, or the ones we sent onsite were as incompetent.
Internal IT is almost invariably a cost center, the technicians providing the service you are selling to customers are working in a profit center. So, yeah, probably that plus be managed in a way which focussed on minimizing cost not maximizing internal customer satisfaction or other quality metrics.
In this case it was a JSON, but using .NET so Newtonsoft is at lease efficient. The issues were many cases of: * Converting string to lowercase to compare them as the keys were case insensitive. (I replaced with a StringComparison.OrderinalCaseInsensitve) * Reduntant Dictionary's * Using Dictionary<Key, Key> as a Set. replaced with HashSet.
The data wasn't meant to be that big, but when a big client was migrated it ended up with 200MB of JSON. (If there data was organised differently it would've been split accross many json blobs instead)
It would also be nice to handle it alls as UTF8 like System.Text.Json does. That would half all the strings saving a fair bit. (I mean the JSON blob it starts with is converted to UTF-16 because .NET)
But I could be wrong. I've only worked with sites and apps not gaming.
I don't think you choose to become a manager at one of the world's top video game companies if you don't love video games.
Hell, I don't think one of the world's top video game companies would hire you as a manager if you didn't convince them, in interviews, that you understand and love their products by using them yourself.
Using your own company's products whenever possible is generally a pretty important part of being a manager.
Same thing happened with memory consumption. The problem was ignored until we regularly had background jobs using > 4 GB of memory, causing cascading failures of our EC2 instances and a whole bunch of strange behaviour as the OOM killer sniped random processes.
Programmer(s): Can we set aside some time to fix the long loading times?
Management: No, that won't earn us any money, focus on adding features
I work LiveOps and usually long loading times are something that we would take seriously, as it negatively impacts the reputation of the game.
I don't think the anti-code-optimization dogma will go away, but good devs already know optimality is multi-dimensional and problem specific, and performance implications are always worth considering. Picking your battles is important, never fighting them nor knowing how is not the trick.
Still, people applying optimizations that sacrifice maintainability for very little gain or increase bugs are still doing a disservice. People who understand data flow and design systems from the get-go that are optimal are where it's at.
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
Yet we should not pass up our opportunities in that critical 3%.
For instance, everyone tells you not to optimize CPU time outside of hotspots, but with memory usage even a brief peak of memory usage in cold code is really bad for system performance because it'll kick everything else out of memory.
Given how incredibly successful it's been, it's conceivable the suits decided the opportunity cost of investing man-hours to fix the issue was too high, and that effort would be better spent elsewhere.
We're talking about an issue that has been loudly complained about for 7 years (and I am evidence that it makes people play the game much less often than they would like to) that some person without access to the source code was able to identify and fix relatively quickly, it would be surprising if it took an FTE 1 week to find this with access to the source code.
This is one of the most profitable games ever, they could hire an entire team to track this down for half a year and it wouldn't even be noticeable on their balance sheet. And I would bet money that it would have increased their active player base (which they care about because of the micro-transactions) in a noticeable way, as long as players were aware of the improvement so they would try it again.
The next step is to optimize away from the cognitively optimal, but only when necessary. So yeah it’s really crazy this was ever a problem at all.
Here the wrong data structure has been used in the first place.
It's just a zero cost one, so if anybody appears complaining that you are caring to choose the correct data structure and that's premature (yeah, it once happened to me too), that person is just stupid. But not calling it an optimization will just add confusion and distrust.
Most of my optimization work is still done with pencil, paper and pocket calculator. Well, actually, most of the time you won't even need a calculator.
See, that's exactly why you're wrong. This wasn't a bad "design". If the fix required to rebuild the whole thing from scratch, then you would have a point and thinking about it "prematurely" would've been a good idea. In this case, both the fixes were bugs that could've been fixed after the game was finished without having to undo much of the work done.
The whole point of the saying is that you don't know what's gonna be a bottleneck until after. Yes by optimizing prematurely, you would've caught those two bugs early, but you would've also spent time analyzing a bunch of other things that didn't need to be analyzed. Whereas if you analyze it at the end once people complain about it being slow, you're only spending the minimum amount of time necessary on fixing the things that matter.
I think that we should also stop doing crash tests in cars. Just release the car to the public and analyze human crashes afterwards.
Premature optimization is about trying to optimize the micro (a given function), while you don't yet have the macro (the game itself). If the function accounts for 0.1% of the performance of the final game, it doesn't matter if you make it 5x faster since that will only make the game 0.08% faster. You could've spent that time optimizing a different function that accounts for 10% of the performance, and even optimizing that function by 5% would make the game 0.5% faster, which is more impactful.
That should be embarrassing for Rockstar but I don't think they would even notice.
Which makes the persisting loading issue all the weirder.
> Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered.
And he is explicitly advocating for optimizing the critical part:
> Yet we should not pass up our opportunities in that critical 3%.
And somehow, people have latched onto the catchphrase about "early optimization" that was taken out of context.
The reason for the misunderstanding is that the kinds of practices it actually talks about are uncommon today. People would often take stuff written in higher level languages and reimplement them in assembler or machine code. Which makes them more time-consuming to change/evolve.
It also isn't like it is hard to figure out which part of a piece of software that is taking up your runtime these days. All worthwhile languages have profilers, these are mostly free, so there is zero excuse for not knowing what to optimize. Heck, it isn't all that uncommon for people to run profiling in production.
Also, it isn't like you can't know ahead of time which bits need to be fast. Usually you have some idea so you will know what to benchmark. Long startup times probably won't kill you, but when they are so long that it becomes an UX issue, it wouldn't have killed them to have a look.
The problem is not that developers can't optimize things: you will find some developers capable of figuring this problem out anywhere. What makes this low hanging fruit so popular is the fact that we aren't measuring enough, and even when we do, we aren't necessarily prioritizing looking into things that are suspiciously slow.
In the case of this example, the issue is also client-side, so it's not as if it's costing CPU time to Rockstar, so it's unlikely you'll have someone who can claim their job description includes wondering if the load times are worth optimizing. When problems like this one get solved is because someone who is very annoyed by the problem and either convinces a developer to even look into the problem. Most of the time, the people that suffer, the people that get to decide how to allocate the time, and the people that are in a position to evaluate the root cause of the problem never even get to talk to each other. It's the price we often pay for specialization and organizations with poor communication.
Organizations where the people deciding what has to be done next, and where the company culture dictates that the way forward is to either complete more tickets faster, or find ways to be liked by your manager, are not going to be fostering the kind of thinking that solves a problem like this one, but that's a lot of what you find in many places. A developer with a full plate that is just working on the next feature isn't going to spend their time wondering about load times.
But instead we end up blaming the developers themselves, instead of the culture that they swim in.
This code looks like someone with almost no experience hacked it together but because they were an intern and likely Rockstar is a toxic place to work, it never gets prioritized to be fixed.
I think if managers prioritized cycle time, metri s more, they'd find that they are encouraging a lot of practices which lead to horrible efficiencies - "measure twice cut once" is a positive mantra which leads to more solid designs with less bugs.
Agile sort of addressed this problem but unfortunately only at small size scales. Iteration and story capacity got overprioritized over quality, customer engagement, and self-leading teams.
Plus things such as scaled agile suffer from the oxymoron of planning fast iteration - if you have a master plan you lose the ability to respond to change and iterate, or you allow iteration and you must accept if any team iterates the whole team must discard the plan...which at some point means you either accept high cycle times or you figure out a way to decouple functionality to the extent the planning becomes the standard fallacy of waterfall - wasting meeting time to go over a plan that isn't based on anything.
This bug wouldn't present in the first couple years with the limited amount of DLC, so by the time it got ridiculous there wasn't anyone left with the confidence to profile the game and optimize it. A junior dev could fix this, but would probably assume that slow loads are a deep complex engine problem that they won't be able to fix.
Alternatively, management would declare that there's too much risk doing more technical engine work, and not sign off on any proposed "minor" optimizations because it's too risky.
GTA Online loading times have been infamous for a very long time. They were already unjustifiably bad when they released the game for PC, and at that point engine programmers would surely be involved.
I'd guess also that the next version of the base engine is in RDR2 or later and doesn't have these issues. But at the same time they likely wouldn't backport the changes for fear of cost overruns.
Our customer portal loads 2 versions of React, Angular, Knockout and jQuery on the same page? Doesn't matter, it's printing billions of dollars.
Rockstar's money printer is so loud that they don't care about problems.
Same thing for Valve, their money printer is so loud that they barely bother to make games anymore and let the Steam client languish for years (how did they let Discord/Twitch happen?).
Not sure that's a fair criticism.
Alyx was widely praised. Artifact... Wasn't. I don't know about Dota Overlords. And that's just the last couple of years.
They've also developed hardware like SteamLink, Steam Controller, some high-end VR gear...
They develop a LOT. They just don't release a while lot.
I agree there should be a lot more work and effort in the client. And they constantly fuck up handling their esports.
But I don't think "barely bother to make games anymore" isn't one of them.
Valve unlike most companies maintains their games for more than 10 years.
Just because they haven't released anything obvious to the casual observer like new single player titles that is easily marketed is only showing your ignorance of Valves entire catalog of games, and attitude to development.
sharepoint is probably many non-engineer's very first exposure to actual version control with checkouts, checkins, version history, and merge.
Like...I don't need it to notify me that I have new e-mail. I already have either Outlook or an Outlook tab running.
Sharepoint is a shining example of feature creep run wild.
Isn't the search issue more of a indexing engine problem though? Can you plug in other engines?
Nowadays, I find Steam Chat is a ghost town.
Their chat system has been famously bad and mostly unchanged since the early 2010's, and only very recently was reworked into this.
Well, I'm using VoIPs basically for >decade everyday and voice quality was never something I cared about (I mean that all soft that I've used was somewhat decent in that matter)
the most important thing is - how to get all people on the same program? and I'm finding it not so realistic to get all friends to Steam
If one really examined the accounting for GTAO, I would bet that most of the billions of dollars that were earned in micro transactions went to marketing, product research, and to middle management in the form of bonuses.
It's bait for the non-discerning customer who is more likely to empty their wallet for microtransactions because they have less experience with games so don't know what is normal :)
Everything is getting slower and slower, and nobody cares.
When I played the Atari 2600, I had to wait for the TV to warm up, but otherwise there were no games with anything approaching load times (with 128 bytes of RAM in the console, who would know). The NES didn't have much in the way of load times either, but you did have to fiddle with the cartridge slot. SNES and Genesis didn't usually load (Air Buster being a noticeable exception). CD based systems sure did like to load, but that's somewhat understandable. In the mean time, more and longer boot screens. The Switch uses a cartridge (or system flash/SD cards), but it likes to load forever too.
PC Gaming has had loading for longer, but it's been getting longer and longer.
Some arcade games have lengthy loading sequences, but only when you turn them on while they read their entire storage into RAM so they can be fast for the rest of the time they're on (which in arcades is usually all day).
It really depends. The latest crop of games I’ve played (Doom Eternal, Cyberpunk) loads way faster than games from a few years back (aforementioned GTA-V, Shadow Warrior 2…).
This is also on the same machine, so it’s not the hardware that makes it faster.
https://devblogs.microsoft.com/directx/directstorage-is-comi...
I think, but I’m not entirely sure, that Linux can do the peer to peer DMA trick. One nasty bit on any OS is that, if a portion of the data being read is cached, then some bookkeeping is needed to maintain coherence, and this adds overhead to IO. I wouldn’t be surprised if consoles had a specific hack to avoid this overhead for read-only game asset volumes.
Also you're probably going to want to do multithreaded decompression anyway, and it'll be more efficient if you have the threads completing the reads do the decompression themselves. So in any case you probably want multiple threads handling the completion events.
With a sufficiently low overhead async API (ie. io_uring) you can saturate a very fast SSD with a single thread, but I'm not sure it would actually make sense for a game engine to do this when it could just as easily have multiple threads independently performing IO with most of it requiring no synchronization between cores/threads.
"Hey we have this great new tech that makes things even faaster!!"
2 years later
"GTA 6 found to have double online load times, denies claims that game performs worse than GTA 5, tells people to upgrade their systems"
4 years later:
"Tech blogger reverses code, realizes someone managed to loop access between hard drive and gpu despite extremely common modern tech, gets 10x boost after spending a day fixing junk product"
Better technology just hasn't met its match from dumber management and more bureaucratic dev shops...
>Everything is getting slower and slower, and nobody cares.
Yes, modern games are so inefficient and laggy nowadays. Once your game world reaches a certain size it becomes unplayable and that's just in single player. Once you add 10 players you start to hit performance limits very quickly.
I can’t recall the name, but I had the hack and slash adventure game variant. The connector on the custom cartridge was fiddly and required a stout rubber band to reliably work.
GTA has achieved tremendous success both as an entertaining game and as a business. It's enjoyed by millions of people and generates billions in revenue. As per this article, it has startup problems (which don't seem to actually really hurt the overall product but I agree sound annoying) but the bigger picture is: it's a huge success.
So - Rockstar has nailed it. What exactly is your platform for analyzing/criticizing their processes or even having a shot of understanding what they are? What have you build that anyone uses? (not saying you haven't, but.. have you been involved with anything remotely close to that scale?)
And if not, whence he high horse?
I and most other customers would argue that 6 minute loading times are atrocious, and if there is an easy fix like this, it makes me lose a lot of respect for the developer who doesn’t fix it. It maybe would even make me avoid them in the future.
A reputation is built over years, but can be lost pretty much instantly. Companies have to continue serving their customers to enjoy ongoing success.
The fact that GTAO is so popular should make most HNers rethink what they know about the commercial necessity of optimization vs building a compelling product.
This really only goes to show how much you can get away with if you have an outstandingly popular product that has no direct competition. Chances are that your product is not that compelling, if it performs poorly, that will hurt adoption. It will never become outstandingly popular in the first place.
They're basically saying that GTAV's massive commercial success should grant it immunity to criticism.
Maybe it got better in recent releases, I kind of stopped following after GTA4.
And yet, no one cares, Those games (and GTA5) all sold millions of copies.
The only way this stuff gets fixed is if (a) some programmer takes pride in load times or (b) customers stop buying games with slow load times.
(b) never happens. If the game itself is good then people put up with the load times. If the game is bad and it has bad load times they'll point to the load times as a reason it's bad but the truth is it's the game itself that's bad because plenty of popular games have bad load times
Also, any game programmer loading and parsing text at runtime by definition, doesn't care about load times. If you want fast load times you setup your data so you can load it directly into memory, fix a few pointers and then use it where it is. If you have to parse text or even parse binary and move things around then you've already failed.
Six minutes is probably excessive, but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game.
Given that it has been the most common criticism of the game since it launched, I don't think anyone views it as a sign of quality.
As things get more complex you're probably going to need to manually set some pointers after loading blobs of data and casting them.
It's just the standard way of dealing with binary files in C. I'm not sure what you'd need for search terms.
As for profiling, Windows Performance Toolkit is the best available no?
You can buy in-game currency for real world money tho: https://gta.fandom.com/wiki/Cash_Cards
Not 100% sure, never bought anything.
Bull. Shit.
There's plenty that can be done because there are parts of a process that don't deserve 15% of the overall budget. The fact that they are taking 1/6 of the time like 5 other things is a failure, not a hallmark of success. Finding 30% worth of improvements with this perspective is easy. 50% often just takes work, but post-discovery much of it is straightforward, if tedious.
My peers are lying with charts to get out of doing "grunt work" when there's a new feature they could be implementing. But performance is a feature.
Having played the game, it's not surprising to me in the least.
I have never yet encountered another such 'wild-west' online experience.
It's the only game that is so un-modereated that i've ever played where the common reaction to meeting a hacker that is interested in griefing you is to call your hacker friend white-knight and ask him to boot the griefer-hacker from the lobby.
Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..)
On top of all this, Rockstar attempts to have an in-game economy , even selling money outright to players in the form of 'Shark Cards' for real-life currency , while 'modders' (hackers) easily dupe gold for anyone that may ask in a public lobby.
This isn't just all coincidence; the game lacks any kind of realistic checks/balances with the server for the sake of latency and interoperability -- but this results in every 13 year old passing around the Cheat Engine structs on game-cheating forums and acting like virtual gods while tossing legitimate players around lobbies like ragdolls -- meanwhile Rockstar continues releasing GTA Online content while ignoring playerbase pleas for supervision.
It's truly unique though -- an online battlefield where one can literally watch battles between the metaphorical white hat and black hat hackers; but it's a definite indicator of a poorly ran business when vigilante customers need to replace customer service.
Also, an aside, most 'mod-menus' -- the small applets put together using publicly available memory structs for game exploit -- most all have a 'quick connect' feature that allows hackers to join lobbies much faster than the GTA V client usually allows for. This feature has existed for years and years, and I believe it performs tricks similar to those listed in the article.
Interesting, back in the day the coolest CTD I did was to simply crank up the combo multiplier in S4 League to crash the game client on every player in the room except mine since that game was peer to peer and thus any form of hacking (teleportation, infinite melee range, instant kill, immortality, etc) was possible. The combo multiplier was set to 256 and thus every single particle was duplicated 256 times and this caused the game to crash.
There's the answer right there. They figure it's making $1B/yr, leave it alone. Maintenance? That cuts into the billion. Everyone moved onto the next project.
When I played it wasn't uncommon to spend 30 minutes mostly looking at the loading screen while you were trying to set up a play session with a couple of friends.
If you're an adult with limited playtime it's just a complete dealbreaker. You can't just decide to have a quick 20minute play session if you know that you'll have to spend at least half of it looking at loading screens.
This oversight has cost them millions of dollars easy.
Probably ranks pretty highly up there in terms of damage to company financials, due to a lack of care.
Ohhh. Thank you for telling me about this. I just found a mirror and successfully built it for macOS. Runs so much better than the wine version. But I guess I'll never finish that RC helicopter mission anyway lol
I think you might be surprised by how few programmers even know what a profiler is, let alone how to run one.
> their parent company unjustly DMCA'd re3
Wow, this is EA games level scumbaggery... I don't think I'm gonna buy games from them again.
Unjustly, but legally. The people you should be salty at are the lawmakers.
I'm impressed that gamecopyworld.com is still online, updated, and has the same UI that it did in 2003
Also
> I don’t think there’s any easier way out.
lmfao
However, games like PUBG and Fortnite are free-to-play (PUBG is only free on mobile?) so in terms of actual sales, you could say GTA is more successful. Still not sure I'd class it as an "online shooter", though.
[1]https://en.wikipedia.org/wiki/List_of_best-selling_video_gam...
I don't know, even though there is shooting in GTA I don't think I'd call it a shooter.
> Not official; the numbers were estimated following a Steam API data leak in July 2018, which gave player ownership estimates for games on the platform that have achievements.
Likely it wasn't fixed precisely because it's such a cash cow. "It's making money, don't fuck with it".
The duplicate checking on the other hand is a classic "accidentally quadratic" case that is obvious.
which of course goes to show that at least from a business side, this issue is completely inconsequential and all resources should be used to push for more monetization (and thus adding to the problem by adding more items to the JSON file) rather than fixing this issue, because, clearly, people don't seem to mind 6 minutes loading time.
I'm being snarky here, yes, but honestly: once you make $1 billion per year with that issue present, do you really think this issue matters at all in reality? Do you think they could make $1+n billion a year with this fixed?
Not surprised they didn't bother for 6 minutes when it takes us 10 years to fix a 30minutes locked startup.
Agree. I found the slow behavior of sscanf while writing one of my first C programs during an internship^^ You literally just have to google "scanf slow" and find lots of information.
Ofc this is just a hypothesis, but I see the hesitation to change legacy code if it ain't broken as a wide spread mentality.
Load times measured in double digit minutes on a significant number of machines meets absolutely every reasonable definition of "broken".
It is both believable and - by virtue of the fact that, as you said, the series continues to be a cash cow - is apparently forgivable.
Here's the thing: the company has zero reasons to fix this, or other ostensibly egregious affronts like DRM, because gamers keep buying the product. There is literally no economic incentive to 'fix' it.
Judging by the number of successful sequels the franchise has spawned, the answer is 'an insignificant number'.
The fact of the matter, and the point my comment was trying to make, is that the overwhelming majority of players do not sufficiently care about load times or other complaints to deter them from paying for the game. That is the reality of the situation.
I'd say especially since the ones who are most likely to be affected by these issues are working adults with limited playtime who won't want to sit in front of their monitor for 5 minutes waiting for the game to load and also happen to be people with disposable income to pour into a game.
(Same thing for websites - they show you all the annoying popup signup sheets/survey questions the instant you load the page.)
"Unbelievable" and "unforgivable" eh? It's a greedy attitude. Instead of viewing GTA5 as a success that's brought a lot of people happiness, you view it as a money cow designed to extract every last bit of profit – and time, since this bug caused 70% longer loading times.
Perhaps it's both. But you, sitting here behind a keyboard with (correct me if I'm wrong) no gamedev experience, have no idea what it's like on a triple-A gamedev team with various priorities. The fact that the game works at all is a minor miracle, given the sheer complexity of the entire codebase.
The fact that someone was able to optimize the obfuscated executable is a wonderful thing. But they weren't a part of the team that shipped GTA 5. If they were, they certainly wouldn't have been able to spend their time on this.
It’s ok to recognize a thing as a business success but a technical failure. In fact many software projects are business successes despite awful and unforgivable quality compromises. You don’t get to whitewash it just because the thing prints money.
This is not small. This kind of incompetency if employed in a different sector such as security would lead to losing personal data of millions.
> “it’s a miracle software works at all”
This is not the case here. Please re-evaluate your calibration on this topic.
2. this kind of incompetence exists in all other sectors. That's why pentests are so crucial, and why they guard the security of millions.
3. we'll have to agree to disagree that it's a minor miracle. Having seen the complexity firsthand, it's quite amazing.
The complexity is in reverse engineering the binary. The developer has access to the full source code and the profiling tools I presume.
Another one is in Microsoft Flight Simulator, instead of downloading multiple archives, it downloads one, unzips it using a single CPU core and then downloads another one. MSFS 2020 takes a few hours to install and that's not just because of the internet connection, but this shitty installation code.
This is what you'd need to decide. And then afterwards, it might not print as much money as you think it will.
It's easy looking at it from the outside. Not so easy from the inside.
I'll meet you halfway though: they should have had profiling sessions that pointed to the JSON parsing code as the issue. I imagine that all of their profiling efforts were focused on the runtime performance, not the load time. Simply put, no one did that profiling, and I don't fault them for focusing on runtime performance (which is where the real money is, as Cyberpunk 2077 demonstrated by not having it, and subsequently having their PS4 orders yanked and refunded).
Once again - Control by Remedy ran on base PS4 at 10 frames per second, TEN frames. Critically acclaimed, reviewers loved it and didnt tend to mention TEN frames per second on base consoles, not pulled from the store.
If this is a serious question, I'd say cut any of the new vehicles introduced in the last 2 years. None of them are nearly as impactful as this optimization. In fact, I am having issues imagine any individual feature at all that's as important as this fix.
Unfortunately this part often kills quality initiatives. Why fix bugs for your existing customers when you can deploy the engineering resources on a DLC or a sequel which will milk those customers for more? There is no more craftsmanship or pride in good work left in software.
"When you're a carpenter making a beautiful chest of drawers, you're not going to use a piece of plywood on the back, even though it faces the wall and nobody will see it. You'll know it's there, so you're going to use a beautiful piece of wood on the back. For you to sleep well at night, the aesthetic, the quality, has to be carried all the way through." -Steve Jobs
How much would it have cost to fix this issue?
Is anyone saying that it is a game developers fault? I mean, what is that you think would prevent a game developer from fixing this?
Because I think, anyone even vaguely familiar with the software industry in general is going to come up with answers like:
1. It would not cost very much 2. No it isn't a developers fault, because it's clear that even an intern could fix this 3. Management isn't interested, or is too disorganized, or focussed on cynical extraction of every last bit of profit.
And from that perspective, it certainly does make it seem like a cynical cash cow.
I don't know many game developers, but I do know people in other parts of the software industry and professionals in general. And I think that they keep to themselves because they have first hand experience of how the industry works and understand it better than anyone. The probably sympathise with the right of the public to feel ripped off.
That said, I still paid for the game, I think it's fun. Apparently there is "no alternative" to this state of affairs.
Well, that's the nominal full retail price against which the various discounts are measured, sure, but I doubt that's what most people buying it these days pay except if they are getting something else with it. I'm pretty sure it's in the stack of things I've gotten free this year on Epic that's in my “I might check it out sometime” queue, it's $29.98 right now from Humble Bundle, etc.
It's actually only $30 and frequently goes on sale for $15. It hasn't been $60 (on Steam at least) since June 2018.
Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.
But, I fully agree with your assessment, for what it's worth.
But a manager shouldn't even have to do that, because in a well functioning team, if the dev leads come back with such a fix, they won't get punished for going off the reservation and would probably be doing this of their own initiative.
But if things like that get met with "why have you wasted time on this? we gave you the list of priorities and it does not include load times" then dev leads will make sure all developers time is filled with things which are prioritized.
Edit: grammar
I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.
This online gamemode alone made $1 billion in 2017 alone.
Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.
Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.
The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.
(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)