News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by liuliu on Friday March 31, 2023 @06:00PM (#35394240)
Metal only recent versions (macOS 13 / iOS 16) supports mmap and use that in GPU directly. CUDA does have unified memory mode even it is dedicated GPU, would be interesting to try that out. Probably going to slow down quite a bit, but still interesting to have that possibility.
[ Parent | on HN ]

There are no comments on this comment yet.