News for hackers. Stuff that matters.
Re:ControlNET and Stable Diffusion: A Game Changer for AI Im... by astrange on Monday February 20, 2023 @01:19AM (#34865500)
It seems to work quite well but StableDiffusion's architecture is quite out of date and the next one (DeepFloyd) may be totally different and not support it.

At least, we know it's capable of generating readable text, which SD sure isn't, and there are newer model papers out there like DiT that should beat latent diffusion on quality and speed.

Re:ControlNET and Stable Diffusion: A Game Changer for AI Im... by ImprobableTruth on Monday February 20, 2023 @07:46AM (#34867581)
Controlnets are just a clever method of finetuning a network to enable additional conditioning. Nothing about it is specific to latent diffusion, so why would another approach like pixel space diffusion not work with it?

Also, I'd hold my horses with calling latent diffusion "out of date". The Imagen paper notes that the results greatly scaled with text-encoder size and the model DeepFloyd (it's a team) are working on uses T5-XXL (the unet is also a decent bit larger, but the Imagen paper claims that this should have a much smaller influence). Latent diffusion might also be able to work with text if you scale up the text encoder.

I'm very skeptical of Google's "trust me, I have the hottest model, it just goes to another school". Muse might have nice fidelity and be fast, but the example images have worse visual quality than Imagen. I definitely wouldn't be surprised if transformers took over, but I don't think it's a foregone conclusion.

Re:ControlNET and Stable Diffusion: A Game Changer for AI Im... by bioemerl on Monday February 20, 2023 @12:47PM (#34870588)
All progress is progress. If your model is better but stable diffusion has all these tools that let you more practically generate the images you want, a whole web UI that's got years of development in it?

Your advanced model becomes a toy, like creating a Twitter clone on your local machine.

You then have to recreate all these things like control net, because I know for sure that this fine tuned control is way more important than a jump in base model quality.

This is why open source kicks ass. The SD people don't have to be the best, not with a community floating them.