News for hackers. Stuff that matters.
Re:SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 40... by flutetornado on Saturday November 09, 2024 @07:15AM (#42093992)
GPU workloads are either compute bound (floating point operations) or memory bound (bytes being transferred across memory hierarchy.)

Quantizing in general helps with the memory bottleneck but does not help in reducing computational costs, so it’s not as useful for improving performance of diffusion models, that’s what it’s saying.

Re:SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 40... by pkAbstract on Saturday November 09, 2024 @07:46AM (#42094140)
Exactly. The smaller bit widths from quantization might marginally decrease the compute required for each operation, but they do not reduce the overall volume of operations. So, the effect of quantization is generally more impactful on memory use than compute.
Re:SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 40... by superkuh on Saturday November 09, 2024 @10:10AM (#42094807)
Except in this case they quantized both the parameters and the activations leading to decreased compute time too.