Re:Stable Diffusion XL technical report [pdf]
by GaggiX on Tuesday July 04, 2023 @07:22PM
(#36593756)
bitsandbytes is only used during training with these models tho (the 8-bit Adamw) quantizing the weights and the activations to a range of 256 values when the model needs to output a range 256 values creates noticeable artifacts as they are not going to map 1-to-1.
[ Parent
| on HN ]
Re:Stable Diffusion XL technical report [pdf]
by liuliu on Tuesday July 04, 2023 @09:19PM
(#36594511)
Draw Things recently released a 8-bit quantized SD model that has comparable output as the FP16. It does use k-means based LUT and separate weights into blocks to minimize quantization errors.
Re:Stable Diffusion XL technical report [pdf]
by GaggiX on Wednesday July 05, 2023 @07:13AM
(#36598680)
I was going to search on the internet about it, but then I realized you are the author (and there is nothing online I think). I imagine that the activations are left in FP16 and the weights are converted in FP16 during inference, right?
Btw very cool
Re:Stable Diffusion XL technical report [pdf]
by liuliu on Wednesday July 05, 2023 @12:22PM
(#36602655)
Yes, computes are carried out in FP16 (so there is no compute efficiency gains, might be latency reductions due to memory-bandwidth saving). These savings are not realized yet because no custom kernels introduced yet.