The Mandalorian filmed a lot of scenes with UE4 doing real-time rendering onto large LED screens like in that demo. Looked pretty good! Probably more fun and easier for the actors too.

Here is how the production looked

https://youtu.be/gUnxzVOs3rk

Incredible. Replace green screen with real time rendered scene. Truman Show IRL.

If you're interested, this was a good article going into details of what's happening behind the scene for the real time rendering: https://ascmag.com/articles/the-mandalorian

We are getting closer to wallpaper and screen meaning the same thing.

That was done for lighting. The lower quality real time imagery was replaced with offline rendered versions in post-production.

"The final animated VFX would be added in later; the screens were merely to provide interactive lighting to match the animations." -- https://ascmag.com/articles/the-mandalorian, discussed here: https://news.ycombinator.com/item?id=22378679

> That was done for lighting.

I don't think that's correct, the ILM team presented this technology in a video:

https://youtu.be/gUnxzVOs3rk?t=104

(paraphrased) "it was designed to both light the actors and be a background that we can photograph, so that we end up with real-time live pixels in-camera"

Also from the article you linked:

> If the content was created in advance of the shoot, then photographing actors, props and set pieces in front of this wall could create final in-camera visual effects — or “near” finals, with only technical fixes required, and with complete creative confidence in the composition and look of the shots.

I'm not an expert in film terminology but it does not sound by any stretch that this was primarily about lighting.

I hadn't seen that video. I wonder how much was just left as the background screens (out of focus backgrounds in closeup shots seem likely). It does make it clear that the ceiling screens were mostly just for lighting, as they don't photograph well from the low angles.

It was for was shot composition as well; the cinematographer could see how it would end up while shooting. The screens updated to match the camera position in real time (e.g. to show the rest of a spaceship whose door was a physical prop).

As mentioned a few times in this comment section, this article covers the subject pretty well: https://ascmag.com/articles/the-mandalorian. All in all, it looks like many shots were truly captured at or near final, without significant post-production modifications.

Yes, I posted that myself :-) It was 2-3 months ago that I read it fully though. Favreau says "A majority of the shots were done completely in camera". However, any shot that included the ceiling screens needed VFX work to cover them, and that alone must have been a lot of shots.

If you can afford to put one or more GPU's per LED screen, there is little limit what you can render real-time, especially when the camera is so far away.

I wonder how many GPU's that system had.

It used four synchronized PCs with a total cost of ~$20,000 to render three 4K panels, which is pocket change for a large production. The LED walls or a single lens costs more than that!

This goes into way more technical details: https://ascmag.com/articles/the-Mandalorian Particularly interesting is how the system had ~10 frames of latency, so excessively fast camera turns would show lower quality renders.

Ah. of course. The system tracked camera and rendered what the camera saw and little around it accurately. Rest was rendered just for lightning.

The guy who developed the tracked camera invented the Roomba (maybe with others? not sure). Pretty cool dude.

I believe I have met him (Burton Conner alum)

> If you can afford to put one or more GPU's per LED screen, there is little limit what you can render real-time

Umm, Ray tracing would love to have a word :P

In all seriousness, typical animated frames for big budget films easily take hours or longer _per frame_. It really depends on what looks you're trying to achieve. Game engines have come a long way in terms of realistic graphics with realtime rendering, but it's worth noting that it's still not the same quality as a fully ray traced scene (whether that matters depends on the content I guess)

A lot of what/why they are doing it this way, is realistic lighting and reflections. Im not sure the difference between a realtime game engine and ray tracing matters that much when you are using it as faux ambient light.

After it is filmed, they can still go back and touchup the backgrounds. Someday with ray tracing they can do real time finished products, but for now the tech works great at what its intended to do.

If a scene has been filmed in this setup, how easy is it to separate the physical foreground from the background screen if they want to re-composite the foreground with a more detailed rey traced background?

The system allows you to insert a dynamic greenscreen around a foreground element (while also retaining the option to preview how things will look after everything). So you can retain most of the virtual set for reflections and lighting, while still having a greenscreen.

I imagine with the advancements in consumer level tech with portrait mode in cameras and zoom backgrounds that the tech ILM has could make easy work of this.

Imagine if the screen flickered at 120hz and half of them were green and the other half were the cgi, and you were able to capture and separate both.

Imagine having to act in such environment... :P

shutter helmet to save you, like active 3d glasses.

And a convenient plot to explain why all characters are using shutter helmets all the time? :)

Yes, of course. But fully animated CGI traces everything back to camera and single screen in movie quality. This setup has 1,326 individual screens, 123,904 px/m² filmed from several meters away for 180 degree view. None of those screens were rendered even close to movie quality.

btw. Only about 50% of the scenes were made with this setup. rest was traditional ray-tracing.

That's true for simple rasterization, but many effects don't scale that way

I wonder if this has anything to do with my wife's complaint (and I agree with her on this) that modern productions are beginning to look more and more like computer games.

She's been watching the third season of westworld, and it's kinda scary to me how almost every episode feels more like a game trailer than an actual story...

Does she watch Game of Thrones? What does she think about those VFX? Or which drama does she think that is not Computer Gaming Graphics like?

It is an interesting complaint. Need some time to think about that in details.

We actually binge watched game of thrones for the first time while we were both on mat leave. Hadn't seen a single episode before then.

The difficulty there is that I think everyone universally acknowledges (?) GOT just fell apart like a car crash in slow motion in the later seasons. It's hard to separate that from the knowledge that they started to depart from the books and were clearly undergoing irreparable problems/pressures by the end of season 5. Don't think we can blame that solely on tech... And it's hard to think about the cgi specifically when the rest is falling apart...

This is really cool, looks like HDR environments in real life. Makes me wonder if this could be used in conjunction with cameras and some clever image processing algorithms as a camouflage method.

Amazing, the future of this tech is just exciting

This is really cool, I just wonder for this particular demo, would it be cheaper to film in a real world location! :)

For movies industry, a lot of the cost are actors and time involved. Having everything filmed with pre made scenes saves you lots of time comparing to green screen back and forth with VFX. Real world location also means lots of travelling.

Traveling with lots of heavy, expensive equipment, setting up, tearing down, then all the unpredictability of outdoor anything (weather, etc).

Also one amazing thing was the guy talking about "shooting a 10-hour dawn" – I can imagine a lot of time is wasted trying to recreate a certain time of day. With this, you don't waste that time. You can do as many takes as you need. The sun stays right where it is.

Absolutely mind-boggling.

> Also one amazing thing was the guy talking about "shooting a 10-hour dawn"

I watched the whole video on the MAnchurian example, and it sold it obviously for me, but I was only initially thinking of the first demo.

Clearly its hugely beneficial.

What I find interesting is, naughty dog had a bit of a staff turnover for designers and hired film animators (in this case it was a crunch), cgi etc.. Funny how they're now almost cross skillset now. Have to admit, makes me feel older every day.

REF: https://www.eurogamer.net/articles/2020-03-14-crunch-once-ag...