Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc...
by petercooper on Wednesday March 15, 2023 @05:21PM
(#35174887)
Can someone a lot smarter than me give a basic explanation as to why something like this can run at a respectable speed on the CPU whereas Stable Diffusion is next to useless on them? (That is to say, 10-100x slower, whereas I have not seen GPU based LLaMA go 10-100x faster than the demo here.) I had assumed there were similar algorithms at play.
[ Parent
| on HN ]
Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc...
by tantony on Wednesday March 15, 2023 @05:31PM
(#35175008)
Stable Diffusion runs pretty fast on Apple Silicon. Not sure if that uses the GPU though.
I think one reason in this particular case may be the 4-bit quantization.
Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc...
by alwayslikethis on Wednesday March 15, 2023 @05:58PM
(#35175306)
Quantization is the answer here. CPU running the large models at 16 bits (which is actually 32, because CPUs mostly do not support FP16) would be really slow.