Re:Real-time image editing using latent consistency models
by orbital-decay on Friday November 10, 2023 @06:03PM
(#38225759)
The architecture is the same but it has an additional model trained on the sampling trajectory computation, speeding up the process. So instead of 20 iterations with the original sampler that solves a SDE, you could use 8, 4, or sometimes even 2 iterations with this new ML-based sampler. That's not a new idea, but the author of this adapter figured out how to train it efficiently using very few GPU-hours.
[ Parent
| on HN ]
Re:Real-time image editing using latent consistency models
by jiggawatts on Friday November 10, 2023 @09:40PM
(#38227156)
I wonder if they could do a real-time preview using 2-4 iterations, and then 'refine' the quality with a higher number of iterations over time, much like how 3D editors preview ray-tracing.
Re:Real-time image editing using latent consistency models
by orbital-decay on Friday November 10, 2023 @10:03PM
(#38227265)
I've lost the link, but IIRC someone on twitter demoed exactly that in Blender. "Neural rendering" in a 3D editor has some nice properties - the scene can be rendered into separate layers with separate depth maps and openpose skeletons generated from the rigs to serve as input for controlnets, objects can be tagged and tags composed into the prompt for the particular tile in case of tiled rendering, etc.
Re:Real-time image editing using latent consistency models
by dvrp on Friday November 10, 2023 @10:47PM
(#38227461)
probably not but are you talking about the guy who used control net and animatediff to make a video that looked relatively consistent?
Re:Real-time image editing using latent consistency models
by orbital-decay on Sunday November 12, 2023 @04:39AM
(#38238776)
No, it was running in the Blender viewport using the original LCM Dreamshaper model.