You're misunderstanding what the parent poster was saying: when working on a large AAA game, content management (art, game design, etc.) is as much a bottleneck as engineering efforts. And a number of those bugs you're concerned about are rooted in content too, not lower level engine bugs.

I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices.

With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.

>I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices.

Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others.

>With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.

There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far.

Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)

Everything you have outlined was standard practice at studios I have worked at for more than five years. The lower in the stack you go, the easier it is to do things like unit tests. In my experience the section of the game industry that is the easiest to test in an automated way is AAA mobile. Where I work in “HD” AAA, it is considerably harder to test in the same ways but we do where it is effective. Don’t mistake the failures of some games or studios as a valid indictment of the industry.

> Don’t mistake the failures of some games or studios as a valid indictment of the industry.

It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.

It's easy if the game engine you are using would display an in-memory model with deterministic algorithms. I have only briefly worked in the game industry and it's been more than enough for me :D. There were like 4-5 people with "10 years+" experience working on a Candy Crush clone. It was literally riddled with bugs and they used Unity's physics engine to make the stones fall (I mean, it doesn't get any more ridiculous than this).

So I came in, throw it all away and built within 2 days a deterministic, fully tested and testable stone engine that would implement all the effects management wanted to see, some of which would have been close to impossible to implement with Unity's physics engine. The idea is basically to use the smallest time unit you want to support, and then base all algorithms on that. You don't work on milli-seconds, because that is for the engine to fill out. You work on seconds, for CandyCrush at least. Each time step is basically a second. The time in-between is filled out by potentially non-deteministic animations and particle effects and whatnot. But every second, the whole scenery will synchronize with the deterministic engine that drives it all.

While I didn't have the chance to try this on shooters or MMORPGs, I think especially for MMORPGs, this would be a perfect fit. It solves sooo many problems, I would probably need a day listing all the benefits. Are there drawbacks? Probably, I can't think of one right now, except that it goes against anything normal game developers believe in.

I think at some point, the gaming industry forked away from solid software engineering practises.

It's a lot more difficult than you make it out to be.

I'm a former game engine lead from an Activision studio. The whole engine architecture was my responsibility, testing included.

You can easily write tests for all the deterministic stuff; math, physics, save games, utility code, filesystem code, etc.

However, where it gets difficult is testing the content. Every platform has some quirks which force you to create art custom made for that platform (your other choice is to use the minimal subset of what they all support). If you want a good looking game, your content has several versions, so now, things like intersection and pathing code are slightly different between platforms. GPU behavior is different enough that using GPU's for anything results in multiple test code paths. AI isn't always fully deterministic, by design, making it even more difficult to test. So, what you have in the end are some deterministic tests to make sure your foundation is sound, and "fuzzy" tests to catch many problems in the higher level constructs, and your final line of defense are play testers, which is an awful job!

Now, we engineers know how to make the games stable and reliable, and our time estimates are frequently at odds with the PM's. You MUST make Christmas, which in the days that I worked, where CD's must be mastered, meant code freeze and asset freeze in October sometime. So, in this mad scramble to make a hard deadline, quality suffered.

It was much more important in the old days (PS1, PS2, GameCube, N64, etc), to get it right the first time, as you couldn't ship patches, but the notion of patching games allowed release versions to be more buggy, since people kicked the can down the road. Granted, the earlier consoles only supported simpler games, and the diffiulty was dealing with the particulars of the system, not the game itself. The PS1 had no Z-Buffer, the PS2 had a CPU/GPU combo built by a crazy person with an PS1 as its IO controller, and the PS3 required you to write DMA management code, while the GameCube had a memory model that was nuts and matrices were only 4x3 so you couldn't do projections. So, you spent most of your time dealing with these quirks and the game ended up simpler due to the effort being spent elsewhere.

Now, it's much easier, consoles are effectively PC's with powerful general purpose CPU's and GPU's, so it should be possible to write high quality, reusable engines.

I've always wondered why big studios don't employ some people that just write bots to play through scenarios etc.

I mean the biggest issue with creating good bots is getting the required data to make decisions such as "you're getting attacked, you need to use this ability" etc, but if you're actually the company writing the game, there shouldn't be any issues creating APIs for that. Or even going directly into memory and read it out, you don't need to worry about getting banned after all.

Imagine Integrationtests in the form of bot scenarios. Is that too much overhead to implement in the usually very budged oriented setting of game development?

You can write a bot player that follows a script or applies some simple decision rules. The hard part is detecting whether or not the game is operating correctly at each step. It's not like a simple web application where things happen in discrete steps and you can inspect the DOM to verify that it contains the right nodes.

Lots of big and small studios have tried to automate game testing. They are interested in it, so if it seems easy and fun you should do it, I’m sure there are jobs out there writing game testing bots.

That said, it would be wise to assume that if you have not heard much about this and wondered why, the answer is probably that it’s harder than you think.

Having been a game lead for a decade, I can safely say that getting game telemetry into the test bots is not the hardest part of the job, it’s one of the easiest things to do. One example of something much more difficult would be testing the game while it changes. Don’t forget that writing a cheat bot for a game that’s done is nothing like testing games that are in development and changing every day.

How would it tell that something is wrong? Like, it went through a wall, and the app let it do so - what’s the problem?

Testing non-deterministic code paths are a really hard problem because you don’t really know what to test for.

Extremely bad take, from somebody who has clearly not worked in AAA games, but believes all the things that gamers post on reddit. A few points:

1) you can only test-driven-develop so much in games, and that line usually stops at the engine level because the game itself is in flux so much. Automated testing is confined to making sure that checkins build on every platform. Game dev engineering is fundamentally different than programming in other fields because the goal posts move constantly.

2) If an engineer writes code expecting there to be no more than 1024 physics objects in the system, tells the designers and artists this, but then they turn around put in 2000 colliding pieces of silverware on a dinner table "because it needs to feel like a big feast" is that an engineering problem, or an art and asset management problem? Because something like 80% of my bugs are shit like this.

3) Professional game codebases use their own styles (i hesitate to say dialects) of C++ that do the things we need them to do the ways we need to do them. We don't use anybody else's framework; all of the bonus stuff we're doing lives in macros that can be inspected if an issue arises. But, please don't push your language purism on anybody else. What a tired argument to have.

None of your points are unique to game development. Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had. There is an eternal back and forth between what the sales team says the software can do, and what the programmers desperately attempt to account for late in the development cycle. Plenty of teams have idiosyncrasies around their tool chain, refuse to reflect on it and maintain that their project is a special case to justify all the mess.

Let me first say that game developers should do a better job about testing, especially when it comes to developing isolated systems to support unit tests. But it certainly seems like you misunderstand how much change there is in the vast majority of game development.

> Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had.

It is hard to describe because it sounds the same. But game development is really different because there are no fundamentals of things you care to test.

Even doing a combat sequence requires a herculean effort. You need basically the whole game running because otherwise what is the point. You fake out most of the data so your test doesn't fail when the designers decide to make combat harder. But now it ends up that pressing A defends instead of attacks because reasons and all of your combat tests now fail.

This is all fixable but it makes the cost per test of anything but the tiniest things hard enough that broad test coverage can sometimes be a determent as you end up testing what the game is now which means it will all be thrown away if your assumptions change.

Sure that can always be the case but "the sum of the lines equals the total" kind of tests are much less likely to backfire in this way.

> Plenty of teams have idiosyncrasies around their tool chain

C++ is chosen because the tooling doesn't exist outside of C++. Full stop. You have to build all the tooling in language X which when talking about 3D graphics is a huge amount of tooling.

Rust is starting to have some cool stuff but if you compare you will see there is a world of difference.

Thus if you choose not C++ you get to write C wrappers around your API and deal with all that nonsense since C++ interop is the worst in most languages.

At some point Rust will get proper C++ interop and then the gap will be smaller but for now you are giving up a ton for a slightly safer language by not choosing C++.

Also note that nearly everybody writes a huge amount of non-C++ code, they just call it a scripting language instead.

A recurring thing that happens in indie dev in the past decade is that some programmer writes up a blog about their "fully test-driven" game.

It always turns out that the game has a trivial state-of-the-art-circa-1980 design with a very small featureset. Nobody doing this is also writing a large RPG or even a Mario style platformer.

So, you can do it, but you spend a massive amount of "scope points" doing it.

The language tooling is much the same way. To do games - big or small - you need lots of I/O handling, and this immediately leads you towards talking to the OS directly, which leads you towards either C or C++ because that's where the tools and resources are. You can get a binding of SDL or whatever for your language, but that's effectively limiting the scope of your engagement with I/O - if the framework doesn't work, you have to debug it across a binding layer which is always iffy. And it can really hurt when talking about console dev.

Offtopic but may I ask you for some game/OpenGL-related questions?

I’m attending a computer graphics course and have to write a minecraft clone - and I don’t yet “feel” the performance of the CPU vs the GPU and sometimes I have trouble deciding which one to push.

Like, am I allowed to call glDraw* multiple times per different objects when the data is already on the gpu, or should I try to push somewhat dissimilar objects into the same buffer and write more complicated shaders differentiating them (without ifs if possible?) or are gpus so performant nowadays that unless I do something stupid I should not worry about it?

> Thus if you choose not C++ you get to write C wrappers around your API and deal with all that nonsense since C++ interop is the worst in most languages.

An example of this is the Swedish game dev company Embark Studios who, after a significant amount of effort, managed to get NVIDIA PhysX (which is widely used in AAA games) working with Rust [1].

[1] https://www.youtube.com/watch?v=RxtXGeDHu0w

You'll have to search very long to find a C++ game code-base that uses boost, game devs are not that stupid ;)

Also, Unity games are usually written in C# (I think it's quite safe to say that - overall - most games are not written in C++ but in C#), yet I've seen no data so far which would indicate that Unity games have any less problems than games written in C++ during production and after release (if anything, the opposite seems to be true, not for technological reasons, but because Unity is so much more beginner-friendly).

I'm no fan of C++ either, but blaming a programming language for bugs and quality problems without any counter examples at hand is a bit ridiculous.

I'm not sure why you say it's a "quite safe" assumption that most games are written in C#. Unity's beginner-friendliness gives it a disproportionate presence online, while the vast majority of AAA games are still solidly C++.

If we eliminate all games with less than 1000 sales or something, I think it would be a very low confidence estimate.

I think perhaps they meant "most Unity games", where Unity was supposed to be implied by the context. It probably is safe to say most Unity games are C#, if we ignore the portion of the engine runtime that is written in C++, which makes sense for some metrics and not others.

> You'll have to search very long to find a C++ game code-base that uses boost, game devs are not that stupid ;)

Don't need to search, factorio used to use boost: https://factorio.com/blog/post/fff-223

The context here, from the top level comment, is AAA game companies and the developers working for them though. In that context, that and indie developer came to the same conclusion after a while is probably just more evidence for the original point.

In a larger studio, even if code sharing isn't common as is expressed here, at a minimum having someone around that sees boost and says "don't do that" is probably a given.

> You'll have to search very long to find a C++ game code-base that uses boost, game devs are not that stupid ;)

Boost has over 160 libraries and counting. I wouldn't recommend every one of them (some have wacky interfaces, slow compile times, and/or experimental designs), but many of them are excellent, and I don't think it's very difficult to tell them apart.

Regardless, I find your insinuation that Boost users are "stupid" to be extraordinarily uncharitable to library users and developers.

> Unity games are usually written in C#

That is true in the most literal sense, but Unity have adopted a programming model that makes much of the normal tooling around .NET useless, and abhors automated testing.

Adopting good testing practices in Unity still largely revolves around developing parts of your game as .NET libraries that get tested before being built and integrated into the game, and it's not too surprising that many developers don't go down this route.

I'm blaming C++ because :

- there is very little information in the community on how to do this kind of engineering efficiently (at least I haven't encountered it nearly as much as I have when I transitioned to application development in higher level languages) - there is very little code sharing in the community and zero standardisation - everyone reinvents shit from standard library, coding conventions, what subset of the language is "allowed", etc. etc. - this means developing good tooling, practices and patterns across large projects is hard

Unity is written in C++, C# is scripting layer and more importantly I doubt Unity developers doing C++ are C# engineers with C# application development background where stuff like automated testing is pretty standard.

> there is very little information in the community on how to do this kind of engineering efficiently

If you aren't a huge studio writing a AAA game or thereabouts in complexity you probably can't write a game engine efficiently.

The number of places that do have that scale and need help on how to be efficient is probably vanishingly small so not a lot of self help pops up.

> (at least I haven't encountered it nearly as much as I have when I transitioned to application development in higher level languages

Pick an engine and use its language whatever that is. Unity gives you C#. Unreal gives you a C++ dialet with some nice features and a blackboard system for scripting that is quite powerful.

> there is very little code sharing in the community and zero standardisation

From my experience there is little code sharing within a studio so expecting there to be code sharing across the entire community is hard.

Assuming you are looking for high performance (AAA) you basically have to write a bespoke engine for your gameplay. And there are a lot of different ways of doing gameplay. Heck whether you have loading zones or continuous loading makes a huge impact on how some fundamental things work (again if you aren't huge there are workarounds that make it easier to generalize in exchange for performance)

> everyone reinvents shit from standard library, coding conventions, what subset of the language is "allowed"

The standard library has some huge performance problems for video games. Mostly around allocation rules. Unsurprisingly algorithms for millions of things can have bad performance side effects on tens of things (and visa versa).

> this means developing good tooling, practices and patterns across large projects is hard

To be fair I think this is generally true. Games just tend to build up their codebases super fast but maintaining any huge code base is a pain in the best cases and it is never the best case.

>I think it's quite safe to say that - overall - most games are not written in C++ but in C#)

Not even close.

> From what I've seen game industry has terrible software engineering practices.

You'd be surprised. Here's a talk from Croteam on how they test their games and engine: https://m.youtube.com/watch?v=YGIvWT-NBHk

I'd wager all major engines are exhaustively tested.

Trouble is, there's combinatorial explosion of game state, user input, assets, scripted behaviour and engine, so there's a huge area to cover.

I mean he's basically saying what I am - they are the exception, very few teams are doing it, no public information on how to do it or best practices, everyone reinvents everything on their own from scratch.

I'd wager the popular engines are well tested because of the number of titles shipped on them not because they have good testing automation - but TBH I haven't worked in this industry for almost 10 years so maybe things changed.

10 years ago was right when "TDD" became hugely hyped. Before that, test automation was patchy throughout the software world, not just in games.

I believe the same is true of "best practice" today: game studios aren't actually behind the curve, you just don't hear much about how things are progressing on this end because most of the conference talks aren't about broad concerns like testing, they're about the myriad specialities of the field.

And there absolutely is a legacy-code thing that hinders AAA in many cases. When the engine is old, that's good, because it's shipped something, but it's bad, because it's using older practices and Things Have Moved On.

That's a fair take too - maybe TDD wasn't as widespread in general so I just got onboard when everyone else did. Although I should note I'm not a fan of TDD and it's not something I would recommend for games or anything similar - it's a very narrow tool - I think you agree because you put "TDD" in quotes I just don't want to make it sound like I'm recommending it - I'm a fan of automated testing.

At least UE4 is not well tested at all. Most of the code doesn't have any decent tests. It's all "developer writes the code and runs once on his machine and then pushes to prod" type of code.

That isn't really true, Unreal uses assertions heavily, see https://docs.unrealengine.com/en-US/ProgrammingAndScripting/...

Unit tests etc aren't that useful in games so no you won't find much of that stuff.

Why would you say something like this?

With everything else being equal Unit tests are as useful for games and game engines as they are for any other type of software. If you care about your software being correct (or let's say less incorrect) you need to test it. And preferably you write automated tests. Games (and especially game engines) aren't really by default an exception to this.

(Of course you can just say that this game is not worth the effort of getting better quality so we don't care if it's a bug fest and that's fine).

I have never found Unit tests particularly useful outside of constrained situations where you have predictable input and output and a initial state that easily reached.

Getting a game(or engine) into a particular initial state so you can make the unit test even work would be a massive pain the ass.

Heavy assertions are more useful as they test actual running code, and they are executed every time the program is run. You can still write "unit test" like code using assertions, but having some code that only executes in the development build on startup to check your math library or whatnot.

Not defending Unreal, I don't particularly like that engine, but that has more to do with the codes age and bloat.

That doesn't sound right. Unit testing happens very close to the unit you want to test, i.e. class or function most of the time. Most of these should be designed with testing in mind so they're actually something you can write unit tests for. And there ought to be a whole bunch of classes/functions/components that can be tested this way, such as scripting code, networking code audio code etc.

It's not hard and definitely not impossible. Just requires a mind set towards quality and testing.

Is that UE4 or games written in UE4? I can totally believe it for games written in UE4, but the engine itself? Doesn't pass the sniff test.

I've been working professionally on a UE4 game with a license to the full UE4 engine codebase, I can confirm that the engine has some unit and functional tests, but the overall coverage is abysmally low.

https://docs.unrealengine.com/en-US/TestingAndOptimization/A... is the overview of the automation system used for unit testing, feature testing, and content stress testing.

There's some built-in testing functionality for the games but there aren't much tests for the engine itself in the source package.

Oh and I spent about 3 years working with UE4. The engine is a bug fest.

> If that was true there wouldn't be so much logic and code bugs in AAA titles.

Those are pretty different concerns tackled by different groups of people on AAA projects. The art requirements can be an order of magnitude larger than the gameplay side of things. That doesn’t make the gameplay side of things easy.

My impression is that asset creation is more scalable (even if labor intensive) and they figured out a way to manage it - I haven't seen a AAA title with asset issues, and from what I understand hiring a bunch of asset creators is cheaper than hiring developers to automate their work.