News for hackers. Stuff that matters.
Re:StableVideo: Text-Driven Consistency-Aware Diffusion Vide... by stale2002 on Monday August 21, 2023 @01:03AM (#37205667)
Did these guys just straight up solve the generative video problem?

The results are better than anything that I've ever seen.

What's the catch? Large processing times? Are the results cherry picked? Or what?

I guess it only works for video to video, but that's still amazing!

Re:StableVideo: Text-Driven Consistency-Aware Diffusion Vide... by satvikpendem on Monday August 21, 2023 @01:11AM (#37205703)
Video to video is not too special, we've had temporally stable solutions to those for some time now, and they're even in commercially available apps. The real test is true text to video, which is much harder.

To my knowledge, the only open source solution that works well for text to video is Zeroscope v2 XL, and v3 is coming soon. v2 is already on par with RunwayML's Gen-2 while v3 is better.

Re:StableVideo: Text-Driven Consistency-Aware Diffusion Vide... by atorodius on Monday August 21, 2023 @01:35AM (#37205802)
Being able to condition on a video vs. just text massively simplifies the task. Mainly because you get consistent camera motion and movement in the scene for free!
Re:StableVideo: Text-Driven Consistency-Aware Diffusion Vide... by progbits on Monday August 21, 2023 @02:18AM (#37206009)
I'm the car example the wheels don't spin which is interesting.

The original frames have different wheel angles so simple text prompted img2img frame by frame approach would preserve the motion, but at the cost of interframe consistency.

Here you get consistent look of the scene and no rapid transitions, but the wheel motion is gone.

Re:StableVideo: Text-Driven Consistency-Aware Diffusion Vide... by remixer-dec on Monday August 21, 2023 @09:39AM (#37209466)
If I understand correctly, you have to train a custom LNA model for each video segment before actually using it.