News for hackers. Stuff that matters.
Re:SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 40... by llm_trw on Saturday November 09, 2024 @07:21AM (#42094016)
Diffusion requires a lot more computation to get results compared to transformers. Naively when I'm running a transformer locally I get about 30% GPU utilization, when I'm running a diffusion model I'm getting 100%.

This means that the only saving you're getting in speed for a diffusion model is being able to do more effective flops since the floats are smaller, e.g. instead of doing one 32bit multiplication, you're doing 8 4bit ones.

By comparison for transformers you not only gain the flop increase, but also the improvement in memory shuffling that they do, e.g. it also takes you 8 times less time to load the memory into working memory from vram.

The above is a vast over simplification and in practice will have more asterisks than you can shake a stick at.

There are no comments on this comment yet.