I strongly doubt there are any GANs involved, they are rarely used in production entertainment, and a solution doesn't really exist for vector geometry afaik. Likely very few scans as well. Just lots and lots of artist hours.

I'm with you that I doubt there are GANs running in real time here, however I don't know see why they wouldn't be using similar models for offline static asset generation.

As far as scans go, it's much cheaper than you would imagine to get high quality face scans. Rigging them for animation is the real challenge, and I'd be really impressed if they were using GANs for rigging.

For state of the art in this space.. check this out: https://twitter.com/ak92501/status/1311832183078879232

Yeah, getting face-scans is cheap and easy. but cleaning them up for real-time rendering, skinning, and rigging as you mention is a huge pain. And then going from there to a generalized model is pretty crazy.

I can imagine they might be using scans that they fit their model to to generate some blendshapes, but it's just as likely that they don't.

Also, nice paper. An initial read of it though suggests it's only capable of building a set of textures and blendshapes, which implies that the actual topology is still in the realm of "artists" We really have no idea how to deal with geometry in ML.

I don't think you can create constraint-based system with 'artist hours'. Just like you can't create realistic 2D face generator without a GAN. It has to be based on samples of real people. There exists lots of generators, such as Daz3D, but you can't create realistic results in these because of their simplistic model. I haven't tested this yet though, so I might be assuming things.

I think you are just assuming things. I think most likely this is exactly like a lot of existing generators, such as Daz3D, but with a lot more work put into them. Especially regarding the material model, hair, animation morphs and the like.

Just look at the rig controls, it's exactly a better, and more polished version of rigging controls that have been used in animation studios for years. I think they are just doing what everyone else is doing, but with just a ton more money.

It’s a sims 4 type character generator, no AI needed.

The companies that Epic acquired to do this do use GANs to create the character. It's not hand crafted. This is a lot of training data feeding an algorithm that decomposes captures down into the constituent parts that you see here.

They aren’t photorealistic but most of them seemed pretty close to my eyes, like movie actors under perfect lighting with a bit too much makeup.

I think they have been close for a few years already. We are edging closer every year but it is still somewhat Digital. Which is sort of the end of the S curve taking a very long time to perfection. May be in 5 years time I can barely tell the difference.

Edit: May be it is the Real Time Rendering.

But the real innovation is achieving this in hours and not weeks or months.

Sometimes I wonder if we are also getting better at distinguishing artificial humans as time goes on. I remember as a kid thinking that G-man in halflife-2 was almost photorealistic.

They need to work on randomizing the left right symmetry a bit in their demos. Humans aren't perfectly symmetrical, and it adds a lot of character to faces.