News for hackers. Stuff that matters.

A comment on Stable Video Diffusion. Parent comment

Re:Stable Video Diffusion by Der_Einzige on Tuesday November 21, 2023 @02:36PM (#38368828)
Controlnet is adapted to video today, the issues are that it's very slow. Haven't you seen the insane quality of videos on civitai?
[ Parent | on HN ]
Re:Stable Video Diffusion by valine on Tuesday November 21, 2023 @02:38PM (#38368870)
I have seen them, the workflows to create those videos are extremely labor intensive. Control net lets you maintain poses between frames, it doesn’t solve the temporal consistency of small details.
[ Parent | on HN ]
Re:Stable Video Diffusion by mattnewton on Tuesday November 21, 2023 @02:43PM (#38368947)
People use animatediff’s motion module (or other models that have cross frame attention layers). Consistency is close to being solved.
[ Parent | on HN ]
Re:Stable Video Diffusion by dragonwriter on Tuesday November 21, 2023 @02:52PM (#38369110)
Temporal consistency is improving, but “close to being solved” is very optimistic.
[ Parent | on HN ]
Re:Stable Video Diffusion by mattnewton on Tuesday November 21, 2023 @03:13PM (#38369399)
No I think we’re actually close. My source is I’m working on this problem and the incredible progress of our tiny 3 person team at drip.art (http://api.drip.art) - we can generate a lot of frames that are consistent, and with interpolation between them, smoothly restyle even long videos. Cross-frame attention works for most cases, it just needs to be scaled up.

And that’s just for diffusion focused approaches like ours. There are probably other techniques from the token flow or nerf family of approaches close to breakout levels of quality, tons of talented researchers working on that too.

[ Parent | on HN ]
Re:Stable Video Diffusion by ryukoposting on Wednesday November 22, 2023 @01:18AM (#38375397)
The demo clips on the site are cool, but when you call it a "solved problem," I'd expect to see panning, rotating, and zooming within a cohesive scene with multiple subjects.
[ Parent | on HN ]
Re:Stable Video Diffusion by mattnewton on Wednesday November 22, 2023 @12:52PM (#38382518)
Thanks for checking it out! We’re certainly not done yet, but much of what you ask is possible or will be soon on the modeling side and we need tools to expose that to a sane workflow in traditional video editors.
[ Parent | on HN ]
Re:Stable Video Diffusion by Hard_Space on Wednesday November 22, 2023 @02:23AM (#38376019)
Once a video can show a person twisting round, and their belt buckle is the same at the end as it was at the start of the turn, it's solved. VFX pipelines need consistency. TC is a long, long way from being solved, except by hitching it to 3DMMs and SMPL models (and even then, the results are not fabulous yet).
[ Parent | on HN ]
Re:Stable Video Diffusion by valine on Tuesday November 21, 2023 @02:50PM (#38369077)
Hopefully this new model will be a step beyond what you can do with animatediff
[ Parent | on HN ]
Re:Stable Video Diffusion by capableweb on Tuesday November 21, 2023 @03:15PM (#38369431)
> Haven't you seen the insane quality of videos on civitai?

I have not, so I went to https://civitai.com/ which I guess is what you're talking about? But I cannot find a single video there, just images and models.

[ Parent | on HN ]
Re:Stable Video Diffusion by capableweb on Wednesday November 22, 2023 @11:08AM (#38381070)
Not sure I'd call that "insane quality", more like neat prototypes. I'm excited where things will be in the future, but clearly it has a long way to go.
[ Parent | on HN ]
Re:Stable Video Diffusion by dragonwriter on Tuesday November 21, 2023 @06:02PM (#38371802)
A small percentage of the images are animations. This id (for obvious reasons) particularly common for images used on the catalog pages for animation-related tools and models, but also its not uncommon for (AnimateDiff-based, mostly) animations to be used to demo the output of other models.
[ Parent | on HN ]
Re:Stable Video Diffusion by adventured on Tuesday November 21, 2023 @06:40PM (#38372251)
https://civitai.com/images

Go there, in the top right of the content area it has two drop-downs: Most Reactions | Filters

Under filters, change the media setting to video.

Civitai has a notoriously poor layout for finding/browsing things unfortunately.

[ Parent | on HN ]