Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc...
by alwayslikethis on Wednesday March 15, 2023 @05:58PM
(#35175306)
Quantization is the answer here. CPU running the large models at 16 bits (which is actually 32, because CPUs mostly do not support FP16) would be really slow.
I think one reason in this particular case may be the 4-bit quantization.