News for hackers. Stuff that matters.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by DrSiemer on Monday June 12, 2023 @02:21AM (#36289126)
Not going to bother with this until the temporal cohesion issue is solved. The results are cool, but the variety in frames makes it look like a very specific and distracting art style instead of true animation.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by colordrops on Monday June 12, 2023 @02:32AM (#36289203)
Did you scroll down and see the different methods used, specifically method 5? Not perfect, but getting pretty close.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by justinclift on Monday June 12, 2023 @02:50AM (#36289329)
It's a lot better than earlier work, but still not really at the level it needs to get to.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by moonchrome on Monday June 12, 2023 @04:33AM (#36290162)
The last one looks the creepiest - like a dystopian matrix glitch. Could be a cool effect to illustrate that.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by rpastuszak on Monday June 12, 2023 @06:24AM (#36291175)
The light reflection on the shoulder doesn't make much sense. I think this is less pronounced when soft light is involved. I wonder if sharp light and reflections will take longer to be handled properly. Every frame on the shoulder looks ok on its own, but when animated it feels like it was taken in a different environment than the rest.

I suspect depth maps (SD2 already supports them) could be used to achieve that in the future.

I wonder if a diffusion model could accept an "onion skin" noise, so the transitions between frames would be less jarring. Can someone with more knowledge than me explain what's the most promising approach here?

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by bondarchuk on Monday June 12, 2023 @06:42AM (#36291360)
>The light reflection on the shoulder doesn't make much sense.

People will really grasp at anything to find something to critique in generative AI, huh?

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by SirMaster on Monday June 12, 2023 @09:13AM (#36292888)
Wait, should we not critique?

OK i'll fix that.

It's perfect. Stop all work and development on it now guys, it's perfect. It has no flaws and makes no mistakes. Amazing I guess.

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by rpastuszak on Monday June 12, 2023 @09:29AM (#36293104)
I think @bondarchuk took my comment out of context, quoting just the first line. No big deal, it's just hard to respond to that.

I don't even see an issue with that particular artefact, I think it's an interesting problem from a technical pov, hence literally every other part of my comment.

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by bondarchuk on Monday June 12, 2023 @09:58AM (#36293505)
It just seemed like a bit of a nitpick in the context of the huge temporal consistency issues/fixes, of course you're all free to discuss whatever you want and I agree it's somewhat interesting.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by rpastuszak on Wednesday June 14, 2023 @02:42AM (#36322283)
Haha, nope, not my intention at all. Although I understand the expectation give the average style of discourse on this site.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by llm_nerd on Monday June 12, 2023 @08:15AM (#36292224)
The last one is largely the original video with a cartoon filter and basically anime facetune. It is educational and interesting, but it looks the best because it's actually doing the least, though even then you get the temporal artifacts on the hair and so on.

It is improving super rapidly and it will be a pretty seamless soon enough in all likelihood. We are in the intermediate stage right now, and the results are going to be incredible.

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by andybak on Monday June 12, 2023 @03:18AM (#36289564)
Mark my words. In a few years people will be writing filters and trying to figure out how to recreate the look of early AI animation.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by raincole on Monday June 12, 2023 @03:38AM (#36289743)
...why? Do you expect GitHub and HuggingFace to go under in a few years?
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by mdp2021 on Monday June 12, 2023 @03:47AM (#36289817)
The poster is just saying that "the "glitch" of today will be the nostalgic style of tomorrow".
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by andybak on Monday June 12, 2023 @04:23AM (#36290072)
Exactly. I have no idea what the other commenters thought I meant.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by theobromananda on Monday June 12, 2023 @06:30AM (#36291235)
They are saying that there is no need to figure out how to recreate "the "glitch" of today that will be the nostalgic style of tomorrow" as the "old"/current models are all available on GitHub and huggingface, and will be in the future.

There is no need to figure out how to recreate this style as you will be able to find it on these platforms in the future. The commenter understood you, but you did not understand the commenter.

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by andybak on Monday June 12, 2023 @06:50AM (#36291418)
Yes. Because of the way reply notifications come in I thought it was a reply to a different comment.

But I still think that's missing the point of my (not entirely serious) comment.

People still have access to analogue amplifiers but digital simulations of them are still developed.

As models and workflows improve, if people want a flickery old-school look then they may well simulate it rather than go through the hassle of running old tools that might not mesh well with newer workflows.

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by staunton on Monday June 12, 2023 @11:39AM (#36295332)
> People still have access to analogue amplifiers but digital simulations of them are still developed.

That's because analog stuff requires hardware. There's "something to simulate". You don't simulate digital computers, perhaps you "emulate" them.

More generally, analog stuff is really different from digital stuff. Some "audiophiles" argue about which lossless compression algorithm applied to .wav files gives the best sound, but we know it's just bits. Bits are fungible. By contrast, you can never make a perfect copy of something analog (you can often make a copy that's indistinguishable for all purposes that matter, but there's still a difference). People still make and buy LPs, the sales are increasing. I don't expect that to ever happen for CDs, unless perhaps combined with scratching or otherwise damaging them on purpose.

Of course, nostalgia goes beyond such mundane distinctions, which gives rise to thingsike pixel art. But there's still nothing to simulate there.

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by deelly on Monday June 12, 2023 @03:52AM (#36289847)
We don't know actually. Could be Cease and Disease letters from creators of AI-networks, could be change in copyright laws, could be simply not profitable enough to continue operations or continue to store old data.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by cfn on Monday June 12, 2023 @03:49AM (#36289828)
Yes, just look at the amount of reviews of old cameras on YouTube! Of course, this is software so it will be easier to find than an old camera or lens but I am also sure it will also be celebrated as a timeless genre.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by DrSiemer on Tuesday June 13, 2023 @08:16AM (#36309118)
For that to happen this flickering animation style has to become popular first. I don't think it will be. It can only be applied to very niche situations. It will never be a mainstream thing like old cameras were.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by downWidOutaFite on Monday June 12, 2023 @04:01AM (#36289904)
Nah, nobody is going to be figuring anything out, you'll be able to generate it with little effort with a prompt, just like everything else. But we will be so inundated by high quality digital art in every imaginable style that it will be boring, everything will be boring.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by dmingod666 on Monday June 12, 2023 @08:39AM (#36292469)
Or they could just use the models we are using right now and get the same effect, not like the models are going away anywhere
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by j45 on Monday June 12, 2023 @08:54AM (#36292643)
Computationally the early models might be easier/cheaper to run in a few years as well.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by objclxt on Monday June 12, 2023 @05:29AM (#36290641)
> the variety in frames makes it look like a very specific and distracting art style instead of true animation.

Well, what is "true animation"?

There are number of types of traditional animation where you draw both the characters and the backgrounds from scratch for every frame, so you get this variety naturally.

Or you have more practical techniques like strata-cut animation[1], where again you get what we might think of today as a very "ML-like" effect of the background constantly shifting.

All of these techniques are very time intensive, so you don't often see them in commercial animation.

[1]:https://en.wikipedia.org/wiki/Strata-cut_animation

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by bondarchuk on Monday June 12, 2023 @06:38AM (#36291325)
In this case we have objects disappearing and reappearing, and the design of clothing and objects (headphones/tiara) being significantly different from frame to frame, pieces of sleeves blinking in and out of existence, headphones becoming tiaras becoming helmets... I'm sure you can find examples of such things in traditional animation but there's clearly something peculiar going on here due to the frames being processed mostly standalone, and it's clearly specific to this exact way of processing. Even if you "draw both the characters and the backgrounds from scratch for every frame", normally the animator would at least have the previous frames and/or overall design at his disposal as a reference. What's happening here is more like those crowdsourcing experiments where you let many people draw a single frame without knowing what the others are up to.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by uwagar on Monday June 12, 2023 @06:47AM (#36291397)
you'd think a real AI would be aware of all these issues and not let this happen.
Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by woctordho on Monday June 12, 2023 @06:54AM (#36291461)
I've had the idea for a long time but never managed to try it: Alias-free convolutions (like StyleGAN3) may help with temporal cohesion.

The StyleGAN3 project page shows some good videos: https://nvlabs.github.io/stylegan3/

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by nialv7 on Monday June 12, 2023 @07:45AM (#36291948)
NVIDIA has already published something that solves the coherence problem, and IIUC it's even based on stable diffusion. Hopefully the community will reproduce it soon.

https://research.nvidia.com/labs/toronto-ai/VideoLDM/

Re:Video to Video with Stable Diffusion (Step-by-Step) – Sta... by beepbooptheory on Monday June 12, 2023 @08:48AM (#36292583)
But isn't also the case you have so much control over the individual frames themselves anyway? Why draw the line where you are? Just based on how these things inherently work, I don't see the specificity problem getting solved anytime soon, short of making a whole new set of weights from some whole new cultural timeline of artistic output.

But its not an issue really! This is just what AI art is. It is something different from just painting: somewhat less and somewhat more in different dimensions. You trade in individual expression and authorship for bricolage and universality. That's cool, but it will always be "specific."

If you just want pretty anime girls or the like, you'd get a lot farther with dumber tools like motion-detection on spine-based animation anyway.