News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by eternalban on Friday March 31, 2023 @05:31PM (#35393942)
Great work. Is the new file format described anywhere? Skimming the issue comments I have a vague sense that r/o matter was colocated somewhere for zero copy mmap or is there more to it?
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Friday March 31, 2023 @07:16PM (#35395072)
That's something I'm working on presently.
[ Parent | on HN ]