Curious about what current SotA is on physics-infusing generation. Anyone have paper links?

OpenAi has a few details:

>> The current model has weaknesses. It may struggle with accurately simulating the physics of a complex scene, and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark.

>> Similar to GPT models, Sora uses a transformer architecture, unlocking superior scaling performance.

>> We represent videos and images as collections of smaller units of data called patches, each of which is akin to a token in GPT. By unifying how we represent data, we can train diffusion transformers on a wider range of visual data than was possible before, spanning different durations, resolutions and aspect ratios.

>> Sora builds on past research in DALL·E and GPT models. It uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for the visual training data. As a result, the model is able to follow the user’s text instructions in the generated video more faithfully.

The implied facts that it understands physics of simple scenes and any instances of cause and effect are impressive!

Although I assume that's been SotA-possible for awhile, and I just hadn't heard?

On the announcement page, it specifically says Sora does not understand physics

I saw similar artifacts in dalle-1 a lot (as if the image was pasted onto geometry). Definitely wouldn't surprise me if they use synthetic rasterized data to in the training, which could totally create artifacts like this.

Wow, yeah I didn't notice it at first, but looking at the rocks in the background is actually nauseating

The water is on par with Avatar. Looks perfect

Maybe it was trained on a bunch of 3d Google Earth videos.

Doesn't look flat to me.

Edit: Here[0] I highlighted a groove in the bushes moving with perfect perspective

[0] https://ibb.co/Y7WFW39

Look in the top left corner, on the plane

The model is essentially doing nothing but dreaming.

I suspect that anything that looks like familiar 3D-rendering limitations is probably a result of the training dataset simply containing a lot of actual 3D-rendered content.

We can't tell a model to dream everything except extra fingers, false perspective, and 3D-rendering compromises.

Technically we can, that's what negative prompting[1] is about. For whatever reason, OpenAI has never exposed this capability in its image models, so it remains an open source exclusive.

[1] https://stable-diffusion-art.com/how-to-use-negative-prompts...

It's more complicated than that. Negative prompts are just as limited as positive prompts.

It's possible it was pre-trained on 3D renderings first, because it's easy to get almost infinite synthetic data that way, and after that they continued the training on real videos.

In the car driving on the mountain road video you can see level-of-detail popping artifacts being reproduced, so I think that's a fair guess.

It looks perfect to me. That's exactly how the area looks in person.

My vote is yes - some sort of intermediate representation is involved. It just seems unbelievable that it's end-to-end with 2D frames...