The cool thing is that this gives game-designers an extremely powerful additional tool. You could for example easily go for a more retro-look by training with kodakcolor images ...
And I don't buy the argument that GTA doesn't look more realistic because that's what the designers were going for. If this was a game with a clear "comic appearance" that might be true, however I would argue it looks like it does, because the designers did the maximum that they could achieve with their tools and within a certain budget.
What they've done is interesting, and I think with further refinement it is certainly a technique to consider.
Impressive result nevertheless. Improving this with adversarial networks will probably generate images which are indistinguishable from reality in just a few years.
Hot take: The overall tone of the style produced using this dataset comes from capture artifacts.
fps > beautiful images until you get at least 60... which this wasn't.
im pretty bearish on this kind of innovation in games tbh, its never going to be low latency enough to be viable, even if they'd improve their framerate - i dont see it going beyond academic interest because of that
definitely interesting to read about though, even if i doubt its viability in gaming
Their post processing can only start after the full image has been rendered, so whatever photorealism filter they apply will necessarily delay the image until the process is finished.
The blurriness, green tinge, and washed out colours are qualities photos sometimes have, but it's not what we see with our eyes. But they do help hide details that would give the image away as a computer rendering.
The creators aren’t necessarily aiming for photorealism. They want their games to look good enough and be entertaining. Adding more realism could make it look worse. Obviously if AI could solve the hard problems like global illumination that’s great. And I’m sure it’ll be possible to e.g easily add more variation to textures by using larger datasets. But changing the tonemapping to a colder palette, or just replacing the asphalt textures with smoother ones?
The creators of GTA didn’t use the sunset tones in the game by some creative mistake or lack of computing power. They obviously went for non-photorealism deliberately! It’s a stylized view of California. It’s like the burger in the commercial doesn’t look like the real one - it looks a bit unrealistically perfect because that’s actually more attractive than the real thing. Realistic and photorealistic are tangentially related.
The comparison here should perhaps be between frames of the game that already attempted to look like the cityscapes dataset.
GTA obviously doesn’t have a ground texture that isn’t smooth because it wouldn’t be possible, or “too warm” palette because it’s too computationally expensive to have the bleak cityscapes tone mapping.
A GTA frame rendered at its current frame cost, with textures the same size (but different) and a customized tone mapping, would look more photorealistic and make the difference to the cityscapes dataset much much smaller.
The authors could likely have done this comparison since they probably have the ability to modify textures and shaders to set tonemapping. That would have separated out the differences that are only artistic from those that are lacking photorealism because of computational expense.
The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.
Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.
RDR2 is based on further development of the GTA5 engine and is arguably the best looking open world game at this time.
They didn't mention using anything else than output frames and G-buffers.
I'd love to be corrected if I'm wrong, BTW.
That's a silly dichotomy. We don't even want films to look completely realistic, that's why CGI exists in the first place. Imagine that you ran this over something like an Avengers movie, or even better, Blade Runner 2049. The AI would remove all the stylized/unrealistic details, and someone here would claim that this is clearly an improvement, and any criticism must come from luddites!
Styling as done in some of them is an optional bonus if creators strive for some additional artistic message, but otherwise its all about money, money, money.
You even contradict yourself - they tried to look Avengers as realistically as possible within the realm of comic book fairy tale, that's why they changed character of Thanos significantly well after introducing him (arguably a good choice). From cartoonish styling to realism. Nobody would take the epic battles so seriously if they all looked like Ready Player One.
Modern CGI in films is already basically indistinguishability from reality. You only notice it these days when it has to be CGI (e.g. space ships) or is badly done (which leads to the impression that CGI is still bad like it was in the 90s).
There are several, several youtube videos showcasing the incredible work of studios like Industrial Light and Magic creating most of the backdrops you thought were shot on location.
Some don’t want games to look this way. Nonetheless this technique can be applied to give games and movies a different look that would very hard to acheive by standard rendering pipelines, and it doesn’t have to look washed out like this example, that’s all based on the training data set and how much this ML lighting and texturing is blended into the scene, and that’s another creative choice. I can imagine filmmakers wanting to adjust CG lighting and texturing of a scene using a training data set from old TV shows to make a convincing reboot of a classic TV series, or new series done in that style.
What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.
This. I thought those were photos they took somewhere that looks exactly the same as GTA V, but then when I watch the video explaining it I was like holy crap. No Ray Tracing or whatever tech demo was ever this close to photo realistic.
>You have no idea how much cheaper this technique is.
Could someone explain this in greater details? I suppose you will still need to create some decent base image or graphics for the enhancement to work to its maximum?
(also, this is simply one guy's opinion, but I am curious if anyone else feels the same)
For instance, in my view, I tend to prefer a stylized less hyper-realism in games. I think this is because the enhanced color, light, and brilliance is more aesthetically pleasant to look at, even though it is certainly more uncommon in the real world.
I don't believe this is because saturated color, and brilliant light doesn't exist in reality, but these occurrences happen more rarely and in specific, often planned, situations and settings. And some people (trained or gifted photographers, for instance), have learned how to capture this look, or to take something rather ordinary, and capture it to appear extraordinary, much like the difference between your average person snapping a photo with a disposable camera, and Ansel Adams. The former will produce a photo that is far more common in appearance, and thus...boring, or uninteresting. So when it comes to games, if you simulate the "average look of reality" the game certainly appears more photo-realistic, but may also be less aesthetically interesting (to me, at least).
This says absolutely nothing about the technology developed here, which is incredibly impressive. Nor do I think there is a right or wrong way create game visuals. I think that misses the point.
However, I feel I'm not the only person who can be blown away with how realistic something can be made to look, yet ...also find it more aesthetically drab and boring to look at. And in the case of games, as I mentioned above, I tend to prefer to see things that are rare, interesting, and extra-ordinary, and thus less "realistic".
Anyone else feel the same?
Without a comparison to the original pictures, they do look very good, the roads in particular look more realistic, as though taken from a dashcam or medium quality phone camera. However the foliage just looks blurred.
A lot of (interactive) CGI already looks more realistic than this. Take for example, this Australian scene made in Unreal Engine 4 - https://www.youtube.com/watch?v=Pg75bfkegtU
And that's not just pre-rendered, here's someone exploring the same scene interactively - https://www.youtube.com/watch?v=y87QL7IPE34
I want my games to look cool. I dont wanna them to look like streetview or geoguessr.
The result is impressive, but I'm not sure it would be good for video games. Maybe with some added filters it would be a lot better.
1. We do not know the set of pictures that the authors did not show us. So maybe this technique works really well only for a small fraction of cases.
2. Games (especially games that involve moving cars) tend to simplify mechanics so much that the moving end result will never look realistic, regardless how perfect the still picture is.
So yes, I expect great outcomes from such a method, but I don't think its going to be a revolutionary silver bullet.
As well, I’m very excited for the scrappy devs who will turn this technique in to something wildly unexpected.
I think this technique has a lot of potential, but will probably need a lot of work before it can be applied to video games. It might first show up as a filter for 'photo mode' in some games though, that would be interesting.
I can't just play GTA 5 as is?
Then, every AAA game looks dumb, non-interactive and limited even with HQ graphics.
I actually don’t think this looks any less “like a video game”, but its color is more washed-out, and darker. It seems like a net loss to me, sorry. Not for fancy theoretical reasons, I just think the GTA looks better in all the examples.
These images are within a hair's breadth of being indistinguishable from reality. It's an enormous leap. An enormous leap from the previous techniques and an enormous leap towards photorealistic realtime graphics.
You don't want your games to look realistic? Go play pacman or outrun.
You want to use path tracing instead? You have no idea how much cheaper this technique is. And on top of that the neural network is also fixing up the unrealistic texture work on all kinds of things in the scene.
The first games to use this are going to be from small game studios with less investment in the status quo and they are going to blow the competition out of the water.