News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by leereeves on Friday March 31, 2023 @08:16PM (#35395565)
> You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use.

How is that possible? Is the model being compressed even more (even after converting to 4 bit) somehow? Or is most of the model unused?

[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by pritambaral on Saturday April 01, 2023 @03:20AM (#35398012)
mmap-ed memory pages backed by a file that aren't dirty aren't counted in an process's RSS usage, only kernel page cache. The mmap-ed regions of virtual memory does get counted in VSZ (virtual memory) but that is just virtual and can be larger than RAM+swap.
[ Parent | on HN ]