Re:Real-time image editing using latent consistency models
by dvrp on Friday November 10, 2023 @10:39PM
(#38227426)
exactly. and there’s more things that you can do in terms of what you use as input for the NN!
Re:Real-time image editing using latent consistency models
by stavros on Saturday November 11, 2023 @04:38AM
(#38228802)
Is it possible to use the seed from the LCM on the full-blown model to get a more detailed image, or is it a different latent space/decoder?
Re:Real-time image editing using latent consistency models
by ttul on Saturday November 11, 2023 @05:13PM
(#38235113)
If you use a tool like ComfyUI, you can use LCM-LoRa to generate fast but not perfect outputs and then refine them with the old school sampler. I’ve been playing around and have found the quality of LCM to be excellent when used in combination with IP Adapter instead of text prompts for conditioning.
From playing around with it a bit locally, LCM is much, much faster, but generally the detail is much lower using the latest SDXL model.
If your prompt is very simple, such as "a boy looking at the moon in a forest" it does pretty well. If your prompt is much more complex and asks for a lot more detail or uses other LoRas, it doesn't do nearly as well as other samplers and generates lower quality, worse matching images. These other samplers take 30-40 steps so it's several times slower.
From what I've seen though, if you use control net, or passing some guideline images in and rely on a simple prompt, like an existing video that you're trying to change the style of, LCM can generate images in near real time like the OP on an RTX4090 and maybe slower cards on smaller/older models
Another benefit is the decreased experimental time. So you can more quickly iterate over seeds to find output you like, and then maybe you can spend some time with other samplers/upscalers with that seed to make the result higher quality