News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by renewiltord on Friday March 31, 2023 @07:32PM (#35395206)
That's the size on disk, my man. When you quantize it to a smaller float size you lose precision on the weights and so the model is smaller. Then here they `mmap` the file and it only needs 6 GiB of RAM!
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by gliptic on Saturday April 01, 2023 @09:33AM (#35400189)
The size mentioned is already quantized (and to integers, not floats). mmap obviously doesn't do any quantization.
[ Parent | on HN ]