Stable Diffusion in pure C/C++

Re:Stable Diffusion in pure C/C++ by api on Saturday August 19, 2023 @09:51AM (#37188792)

It’s interesting to me that my CPU can run some of these things in quantized form almost as fast as the GPU. Has the whole thing been all about memory bandwidth all along?

In addition to compute the GPU architecture is one that somewhat colocates working memory alongside compute. Units have local memories that sync with global memory. Is that a big part of why GPUs are so good for this?

[ Parent | on HN ]

Re:Stable Diffusion in pure C/C++ by brucethemoose2 on Saturday August 19, 2023 @10:45AM (#37189327)

> Has the whole thing been all about memory bandwidth all along

Yeah, sort of.

LLMs like llama at a batch size of 1 are hilariously bandwidth bound.

Stable Diffusion less so. Its still bandwidth heavy on GPUs, but compute is much more of a bottleneck.

[ Parent | on HN ]