News for hackers. Stuff that matters.
Re:Enhancing Photorealism Enhancement by plutonorm on Wednesday May 12, 2021 @05:18AM (#27128315)
The number of comments here that are negative is just staggering to me.

These images are within a hair's breadth of being indistinguishable from reality. It's an enormous leap. An enormous leap from the previous techniques and an enormous leap towards photorealistic realtime graphics.

You don't want your games to look realistic? Go play pacman or outrun.

You want to use path tracing instead? You have no idea how much cheaper this technique is. And on top of that the neural network is also fixing up the unrealistic texture work on all kinds of things in the scene.

The first games to use this are going to be from small game studios with less investment in the status quo and they are going to blow the competition out of the water.

Re:Enhancing Photorealism Enhancement by cycomanic on Wednesday May 12, 2021 @06:18AM (#27128647)
I completely agree with you. Also those people who complain about the "dull european" look, did you scroll down and look at the images altered using the Mapillary Vistas set?

The cool thing is that this gives game-designers an extremely powerful additional tool. You could for example easily go for a more retro-look by training with kodakcolor images ...

And I don't buy the argument that GTA doesn't look more realistic because that's what the designers were going for. If this was a game with a clear "comic appearance" that might be true, however I would argue it looks like it does, because the designers did the maximum that they could achieve with their tools and within a certain budget.

Re:Enhancing Photorealism Enhancement by tallanvor on Wednesday May 12, 2021 @06:48AM (#27128785)
I almost clicked away from the the article because those Cityscapes transformations looked worse in my opinion - just washed out and dull, so I can understand why people didn't like the look. The Mapillary Vistas processing definitely was nicer, although it looked uneven to me - the road still looked washed out, which created an almost disturbing level of contrast with the greenery.

What they've done is interesting, and I think with further refinement it is certainly a technique to consider.

Re:Enhancing Photorealism Enhancement by marcodiego on Wednesday May 12, 2021 @08:21AM (#27129428)
I think expectations influenced by culture play a role here. Since I live in a Brazilian metropolis (4mi population), the Cityscapes transformations looked more real. That is probably because they look more like what I'm used to see when I walk the streets. People from rich neighborhoods in the US will likely find the Mapillary Vistas more realistic since that is what they used to see.

Impressive result nevertheless. Improving this with adversarial networks will probably generate images which are indistinguishable from reality in just a few years.

Re:Enhancing Photorealism Enhancement by formerly_proven on Wednesday May 12, 2021 @08:35AM (#27129538)
The color grade in GTA is of course an artistic choice and rendering the entire game with foggy overcast light and a strong green tinge (which btw. is not how German cities look, we aren't living in a bad color transfer of The Matrix) makes it look bad at the macro level. The details might be enhanced by this method, but honestly it's kind of hard to tell from a heavily compressed 720p Youtube video. What looks okay or even better in a video doesn't have to look better interactively, uncompressed, in native resolution.

Hot take: The overall tone of the style produced using this dataset comes from capture artifacts.

Re:Enhancing Photorealism Enhancement by 411111111111111 on Wednesday May 12, 2021 @09:36AM (#27130146)
the only thing i could get out of the demo videos is that their fps is shit.

fps > beautiful images until you get at least 60... which this wasn't.

im pretty bearish on this kind of innovation in games tbh, its never going to be low latency enough to be viable, even if they'd improve their framerate - i dont see it going beyond academic interest because of that

definitely interesting to read about though, even if i doubt its viability in gaming

Re:Enhancing Photorealism Enhancement by container on Thursday May 13, 2021 @04:09AM (#27139641)
>its never going to be low latency enough to be viable Why is that? Just because it's expensive in terms of computation? I can't see why latency wouldn't improve with framerate.
Re:Enhancing Photorealism Enhancement by 411111111111111 on Thursday May 13, 2021 @07:59PM (#27149038)
It couldn't because of the nature of this technology.

Their post processing can only start after the full image has been rendered, so whatever photorealism filter they apply will necessarily delay the image until the process is finished.

Re:Enhancing Photorealism Enhancement by Scea91 on Friday May 14, 2021 @01:41PM (#27157252)
This is not a fundamental obstacle for the technology. Lots of stuff is happening in post-processing already.
Re:Enhancing Photorealism Enhancement by nearbuy on Wednesday May 12, 2021 @03:25PM (#27134168)
The Cityscapes transformations have a photorealism advantage because they're blurrier and more washed out. It's easier to match less detailed photos.

The blurriness, green tinge, and washed out colours are qualities photos sometimes have, but it's not what we see with our eyes. But they do help hide details that would give the image away as a computer rendering.

Re:Enhancing Photorealism Enhancement by Cthulhu_ on Wednesday May 12, 2021 @08:15AM (#27129396)
Definitely; I mean if they had a dataset of the cities and locations GTAV was based on I think it might look uncanny.
Re:Enhancing Photorealism Enhancement by alkonaut on Wednesday May 12, 2021 @05:45AM (#27128470)
> You don't want your games to look realistic?

The creators aren’t necessarily aiming for photorealism. They want their games to look good enough and be entertaining. Adding more realism could make it look worse. Obviously if AI could solve the hard problems like global illumination that’s great. And I’m sure it’ll be possible to e.g easily add more variation to textures by using larger datasets. But changing the tonemapping to a colder palette, or just replacing the asphalt textures with smoother ones?

The creators of GTA didn’t use the sunset tones in the game by some creative mistake or lack of computing power. They obviously went for non-photorealism deliberately! It’s a stylized view of California. It’s like the burger in the commercial doesn’t look like the real one - it looks a bit unrealistically perfect because that’s actually more attractive than the real thing. Realistic and photorealistic are tangentially related.

The comparison here should perhaps be between frames of the game that already attempted to look like the cityscapes dataset.

GTA obviously doesn’t have a ground texture that isn’t smooth because it wouldn’t be possible, or “too warm” palette because it’s too computationally expensive to have the bleak cityscapes tone mapping.

A GTA frame rendered at its current frame cost, with textures the same size (but different) and a customized tone mapping, would look more photorealistic and make the difference to the cityscapes dataset much much smaller.

The authors could likely have done this comparison since they probably have the ability to modify textures and shaders to set tonemapping. That would have separated out the differences that are only artistic from those that are lacking photorealism because of computational expense.

Re:Enhancing Photorealism Enhancement by dplavery92 on Wednesday May 12, 2021 @12:51PM (#27132405)
I am not very interested in whether or not the dataset that these authors chose is a pleasing dataset to use as a basis for a video game aesthetic. I am interested in in the technical accomplishments that this model was able to achieve in regards to preserving image features of semantic importance (e.g. writing, symbols, logos), not "hallucinating" new image features, and maintaining frame-to-frame stability in the output video product. These are very notable accomplishments for an adversarial model in this space, and the techniques they used to accomplish them are interesting. That Rockstar could have made their game more Cityscapes-like without this tech is missing the forest for the trees.
Re:Enhancing Photorealism Enhancement by kaibee on Wednesday May 12, 2021 @01:04PM (#27132574)
Honestly my main take here is just how much better the foliage can look. Grass/trees in real time graphics just aren't even close.
Re:Enhancing Photorealism Enhancement by bayindirh on Wednesday May 12, 2021 @05:34AM (#27128403)
Actually, I'm pretty amazed by the results. This is indeed a giant leap. Also it's coming from Intel instead of AMD or Nvidia, which makes me hopeful about its wide availability.

The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.

Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.

Re:Enhancing Photorealism Enhancement by formerly_proven on Wednesday May 12, 2021 @08:37AM (#27129554)
> The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.

RDR2 is based on further development of the GTA5 engine and is arguably the best looking open world game at this time.

Re:Enhancing Photorealism Enhancement by garaetjjte on Wednesday May 12, 2021 @12:31PM (#27132188)
>They've probably grafted a lot of tapping points to video driver to be able to implement this.

They didn't mention using anything else than output frames and G-buffers.

Re:Enhancing Photorealism Enhancement by bayindirh on Wednesday May 12, 2021 @01:39PM (#27133002)
I'm not an expert in graphics programming but I don't think that accessing G-Buffers of an application and grafting your ML based post processor into the pipeline is exactly trivial. From my point of view it's only possible by either engine provided tapping points, or you tap the processing pipeline at the driver level directly.

I'd love to be corrected if I'm wrong, BTW.

Re:Enhancing Photorealism Enhancement by garaetjjte on Wednesday May 12, 2021 @01:58PM (#27133246)
You need to hook into game process, but it's nothing too hard. Either patching game binary in right place to funnel pointer to needed buffers to your DLL, or hooking something like D3D11 OMSetRenderTargets and doing some clever guessing which one of buffers this is. Doing this at driver side wouldn't make any difference, as driver doesn't have any extra semantic info what the buffer is used for anyway.
Re:Enhancing Photorealism Enhancement by codeflo on Wednesday May 12, 2021 @06:28AM (#27128691)
> You don't want your games to look realistic? Go play pacman or outrun.

That's a silly dichotomy. We don't even want films to look completely realistic, that's why CGI exists in the first place. Imagine that you ran this over something like an Avengers movie, or even better, Blade Runner 2049. The AI would remove all the stylized/unrealistic details, and someone here would claim that this is clearly an improvement, and any criticism must come from luddites!

Re:Enhancing Photorealism Enhancement by saiya-jin on Wednesday May 12, 2021 @07:09AM (#27128927)
The primary reason why CGI is used is to save money. The secondary is to ease production of movies, which at the end saves money and time (which is money).

Styling as done in some of them is an optional bonus if creators strive for some additional artistic message, but otherwise its all about money, money, money.

You even contradict yourself - they tried to look Avengers as realistically as possible within the realm of comic book fairy tale, that's why they changed character of Thanos significantly well after introducing him (arguably a good choice). From cartoonish styling to realism. Nobody would take the epic battles so seriously if they all looked like Ready Player One.

Re:Enhancing Photorealism Enhancement by klmadfejno on Wednesday May 12, 2021 @08:57AM (#27129778)
If I want to film a dragon, I'm probably going to choose CGI because it gets me the best dragon. It is coincidentally cheaper than building a puppet, or genetically engineering a lizard to look like a dragon, but cost isn't the motivating factor here.
Re:Enhancing Photorealism Enhancement by IshKebab on Thursday May 13, 2021 @03:55PM (#27146586)
What? That's not at all why CGI exists. It's to save money and increase flexibility, and sometimes to do shots that would otherwise be impossible.

Modern CGI in films is already basically indistinguishability from reality. You only notice it these days when it has to be CGI (e.g. space ships) or is badly done (which leads to the impression that CGI is still bad like it was in the 90s).

Re:Enhancing Photorealism Enhancement by Hard_Space on Wednesday May 12, 2021 @09:13AM (#27129951)
The only director who laboriously follows through on 'I want it to not look like CGI' is Christopher Nolan. Others seem to favor stylized beauty over absolute CGI realism.
Re:Enhancing Photorealism Enhancement by thenoxx on Wednesday May 12, 2021 @11:45AM (#27131649)
Uh, no, you just don't notice the absolute gobs of CGI used in tons of films because it is meant to look hyperrealistic.

There are several, several youtube videos showcasing the incredible work of studios like Industrial Light and Magic creating most of the backdrops you thought were shot on location.

Re:Enhancing Photorealism Enhancement by snorkel on Wednesday May 12, 2021 @07:32AM (#27129078)
This is an enormous leap. Texturing and lighting are two very hard problems in CGI. Realistic lighting using raytracing requires a lot of radiosity passes, which means adjusting lighting of each surface to account for each ray of light bouncing from surface to surface to surface to surface like a 3D billiard ball, more bounces computed yields better results but there’s computational limits to the number of reflections per ray, and many parameters and formulas to choose from each with its own trade offs. Texturing and shader pipelines is another matrix of trade offs to get a good-enough result. Instead of all of that complexity, applying ML this way cleverly hops over the most difficult parts to a really impressive result.

Some don’t want games to look this way. Nonetheless this technique can be applied to give games and movies a different look that would very hard to acheive by standard rendering pipelines, and it doesn’t have to look washed out like this example, that’s all based on the training data set and how much this ML lighting and texturing is blended into the scene, and that’s another creative choice. I can imagine filmmakers wanting to adjust CG lighting and texturing of a scene using a training data set from old TV shows to make a convincing reboot of a classic TV series, or new series done in that style.

Re:Enhancing Photorealism Enhancement by HellDunkel on Wednesday May 12, 2021 @05:31AM (#27128385)
Maching learning is often oversold as a shortcut solution for some very hard problems and this is a perfect example.

What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.

Re:Enhancing Photorealism Enhancement by plutonorm on Wednesday May 12, 2021 @05:52AM (#27128510)
So were ajax crud web apps, 25 years ago.
Re:Enhancing Photorealism Enhancement by zokier on Wednesday May 12, 2021 @06:19AM (#27128648)
And they still suck, 25 years later
Re:Enhancing Photorealism Enhancement by ksec on Wednesday May 12, 2021 @08:56AM (#27129756)
>These images are within a hair's breadth of being indistinguishable from reality.

This. I thought those were photos they took somewhere that looks exactly the same as GTA V, but then when I watch the video explaining it I was like holy crap. No Ray Tracing or whatever tech demo was ever this close to photo realistic.

>You have no idea how much cheaper this technique is.

Could someone explain this in greater details? I suppose you will still need to create some decent base image or graphics for the enhancement to work to its maximum?

Re:Enhancing Photorealism Enhancement by 0-_-0 on Wednesday May 12, 2021 @09:00AM (#27129810)
I think the source game could be lower polygon count and texture resolution, but the neural network might be quite heavy...
Re:Enhancing Photorealism Enhancement by andrew-dc on Wednesday May 12, 2021 @11:25AM (#27131413)
I think photo-realism in games is not merely a black and white, positive or negative. There are trade-offs, and I think it has to do with perception around aesthetics.

(also, this is simply one guy's opinion, but I am curious if anyone else feels the same)

For instance, in my view, I tend to prefer a stylized less hyper-realism in games. I think this is because the enhanced color, light, and brilliance is more aesthetically pleasant to look at, even though it is certainly more uncommon in the real world.

I don't believe this is because saturated color, and brilliant light doesn't exist in reality, but these occurrences happen more rarely and in specific, often planned, situations and settings. And some people (trained or gifted photographers, for instance), have learned how to capture this look, or to take something rather ordinary, and capture it to appear extraordinary, much like the difference between your average person snapping a photo with a disposable camera, and Ansel Adams. The former will produce a photo that is far more common in appearance, and thus...boring, or uninteresting. So when it comes to games, if you simulate the "average look of reality" the game certainly appears more photo-realistic, but may also be less aesthetically interesting (to me, at least).

This says absolutely nothing about the technology developed here, which is incredibly impressive. Nor do I think there is a right or wrong way create game visuals. I think that misses the point.

However, I feel I'm not the only person who can be blown away with how realistic something can be made to look, yet ...also find it more aesthetically drab and boring to look at. And in the case of games, as I mentioned above, I tend to prefer to see things that are rare, interesting, and extra-ordinary, and thus less "realistic".

Anyone else feel the same?

Re:Enhancing Photorealism Enhancement by ryankrage77 on Wednesday May 12, 2021 @01:46PM (#27133095)
> These images are within a hair's breadth of being indistinguishable from reality

Without a comparison to the original pictures, they do look very good, the roads in particular look more realistic, as though taken from a dashcam or medium quality phone camera. However the foliage just looks blurred.

A lot of (interactive) CGI already looks more realistic than this. Take for example, this Australian scene made in Unreal Engine 4 - https://www.youtube.com/watch?v=Pg75bfkegtU

And that's not just pre-rendered, here's someone exploring the same scene interactively - https://www.youtube.com/watch?v=y87QL7IPE34

Re:Enhancing Photorealism Enhancement by plutonorm on Thursday May 13, 2021 @05:41AM (#27140312)
This is because its from scanned assets, I.e. there's no artist trying to imitate reality. The neural network above can take your artist created graphics and make them appear like they were scanned in. Both techniques achieve the same thing.
Re:Enhancing Photorealism Enhancement by malka on Wednesday May 12, 2021 @12:03PM (#27131866)
> You don't want your games to look realistic? Go play pacman or outrun.

I want my games to look cool. I dont wanna them to look like streetview or geoguessr.

The result is impressive, but I'm not sure it would be good for video games. Maybe with some added filters it would be a lot better.

Re:Enhancing Photorealism Enhancement by choeger on Wednesday May 12, 2021 @09:27AM (#27130048)
While I agree that the quality of this method is impressive, there are two caveats:

1. We do not know the set of pictures that the authors did not show us. So maybe this technique works really well only for a small fraction of cases.

2. Games (especially games that involve moving cars) tend to simplify mechanics so much that the moving end result will never look realistic, regardless how perfect the still picture is.

So yes, I expect great outcomes from such a method, but I don't think its going to be a revolutionary silver bullet.

Re:Enhancing Photorealism Enhancement by joshspankit on Thursday May 13, 2021 @05:02PM (#27147422)
Not only am I extremely impressed with the current realism they were able to accomplish, I suspect that others will figure out what it is about the final result and that will feed back in to even further realism in games (just like the breakthrough of subsurface scattering in digital skin).

As well, I’m very excited for the scrappy devs who will turn this technique in to something wildly unexpected.

Re:Enhancing Photorealism Enhancement by solotronics on Wednesday May 12, 2021 @10:08AM (#27130497)
I thought the video was from real life at first until I listened to the explanation. It's amazing that this is possible in real time.
Re:Enhancing Photorealism Enhancement by Cthulhu_ on Wednesday May 12, 2021 @08:18AM (#27129413)
Could they do this process at video game speeds though? (25-60 times a second)

I think this technique has a lot of potential, but will probably need a lot of work before it can be applied to video games. It might first show up as a filter for 'photo mode' in some games though, that would be interesting.

Re:Enhancing Photorealism Enhancement by cecilpl2 on Wednesday May 12, 2021 @11:01AM (#27131147)
The first video on that page says they can run it at interactive framerates.
Re:Enhancing Photorealism Enhancement by ricklamers on Thursday May 13, 2021 @01:15PM (#27144645)
I think you’re absolutely right. The innovations in deep learning make it such an exciting time to be an indie game developer. What used to take armies of content creators can now be a tiny team. Unity should be all over this kind of tech.
Re:Enhancing Photorealism Enhancement by kfarr on Wednesday May 12, 2021 @11:50AM (#27131712)
Yeah the foliage is remarkably life-like and that is so hard and labor (art) intensive to do well. This is an amazing tool for indies without tons of resources.
Re:Enhancing Photorealism Enhancement by armageddon on Wednesday May 12, 2021 @11:49AM (#27131710)
> You don't want your games to look realistic? Go play pacman or outrun.

I can't just play GTA 5 as is?

Re:Enhancing Photorealism Enhancement by amelius on Wednesday May 12, 2021 @07:39AM (#27129130)
Photorealism shouldn't be the end goal. Watch a B/W television for a week and you won't be missing the colors. Photorealism is much like sugar in that it is addictive, but you won't miss the sugar if you didn't eat it for a while.
Re:Enhancing Photorealism Enhancement by Cthulhu_ on Wednesday May 12, 2021 @08:17AM (#27129409)
It's not the end goal, but it's definitely an aesthetic direction that a lot of games want to go towards. And not just video games, but films too. Most films already have photorealistic CGI, sometimes applying it without you even noticing it (e.g. backgrounds).
Re:Enhancing Photorealism Enhancement by anthk on Wednesday May 12, 2021 @08:32AM (#27129506)
Better, play IF games/roguelikes such as Slashem/DCSS, Gearhead, Anchorhead, Vicyous Cycles, Slouch over Bedlam and All Thing Devours.

Then, every AAA game looks dumb, non-interactive and limited even with HQ graphics.

Re:Enhancing Photorealism Enhancement by klmadfejno on Wednesday May 12, 2021 @09:01AM (#27129813)
And yet, the vast majority of DCSS players choose tiles over ASCII. Liking games without nice visuals doesn't mean people don't like nice visuals.
Re:Enhancing Photorealism Enhancement by draw_down on Wednesday May 12, 2021 @11:07AM (#27131225)
Whenever there is a sanctimonious comment like this at the top complaining about all the negative posts, I usually agree with the other posts. As in this case.

I actually don’t think this looks any less “like a video game”, but its color is more washed-out, and darker. It seems like a net loss to me, sorry. Not for fancy theoretical reasons, I just think the GTA looks better in all the examples.