News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by shakow on Friday March 31, 2023 @08:12PM (#35395525)
Disk accesses should not lie. If only 6GiB are read from the disk, then I believe either the model is indeed sparse in its computation, or there may be a bug somewhere.
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by outofpaper on Friday March 31, 2023 @08:52PM (#35395843)
You couldn't have said it clearer.
[ Parent | on HN ]