News for hackers. Stuff that matters.
Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc... by alwayslikethis on Wednesday March 15, 2023 @05:58PM (#35175306)
Quantization is the answer here. CPU running the large models at 16 bits (which is actually 32, because CPUs mostly do not support FP16) would be really slow.

There are no comments on this comment yet.