News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by gliptic on Saturday April 01, 2023 @09:33AM (#35400189)
The size mentioned is already quantized (and to integers, not floats). mmap obviously doesn't do any quantization.
[ Parent | on HN ]

There are no comments on this comment yet.