Predicting the next frame from previous frames and learnt sequences is a neat trick, but I think automated game development is already very possible with simple techniques like genetic algorithms, or even a PRNG. I mean Rogue and it's descendants are very much automated game development, but GPT-3 could be useful for something like dynamic quest generation, world-building, narrative, adaptive NPCs (including dialogue) etc.

I'm not sure classic procedural content generation like in rogue-likes is all that comparable to using a GAN to run the whole game?

Have a look at the video, it's quite impressive.

I've seen it before, and I've a basic understanding of GANs, I just don't see it being overly useful. This technique can make a really blurry simulacrum of an actual game, and that's really cool, but I'm not sure how it could be used to make a something both truly novel and coherent. There's plenty of low hanging fruit for AI within an engine, whereas using AI to be the entire engine is somewhat infeasible

You are right about the technique not being very useful as of today. My fascination stems from my assumption that more resources poured into this approach would yield vastly better results.

Even just watching the video, I came up with several possible improvements to try out. Eg adversarial training, that would really hone in on the situations and aspects where the model is weak so far, like edge conditions; instead of just using normal gameplay as input.

It's definitely an interesting area of research, but for that example you still had to make the whole game in the first place in order to have something to train the model on. Say you have a novel game idea, how could you use that approach to make it a reality? I'm not sure you could, but like you mention it's a really early example and who knows where it ends up.

The other part about that GAN Theft Auto example is that it doesn't actually know what's going on, like there's no game state. All it knows is that "When I have a frame that looks like this, and they press that button, I think the next frame would usually look like this". So it's got no internal game logic, it's just really good at painting what games look like.

About the first one:

Even going about this very naively, you could at least use it to train a model against a supercomputer running the game, and then run the inference on much more modest end-user machines.

But you can be much more ambitious: have you seen eg style transfer? So you could probably do a bit of ML black magic to train your model on GTA, and then point it at the Google Earth data to get a GTA-like set in real-life London.

Or you could use something like style transfer to go for a cartoony look, or add ray-tracing like effects, even if you didn't have these effects in your original engine.

Or you can use a pre-trained model (eg on GTA), and then spend a relatively modest amount of extra training to get a different kind of game, eg one that has magic or so.

About the latter part: I do think their model is already running with some state. But even if it ain't, that's a relatively small thing to add with already known standard techniques (or you can come up with new techniques.)