Llama.cpp 30B runs with only 6GB of RAM now

Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Friday March 31, 2023 @06:07PM (#35394335)

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by conradev on Friday March 31, 2023 @06:15PM (#35394430)

Very cool! Are you testing after a reboot / with an empty page cache?

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Friday March 31, 2023 @06:38PM (#35394709)

* * *

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by dekhn on Saturday April 01, 2023 @11:01AM (#35400835)

Linux has a command to drop caches at runtime (https://www.tecmint.com/clear-ram-memory-cache-buffer-and-sw...) which is VERY useful during debugging.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by Miraste on Friday March 31, 2023 @06:47PM (#35394797)

This is incredible, great work. Have you tried it with the 65B model? Previously I didn't have a machine that could run it. I'd love to know the numbers on that one.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by startupsfail on Friday March 31, 2023 @07:48PM (#35395345)

Hmm, can you try running iotop, to see how much is being red from disk? Is it 20Gb or 6Gb? Maybe the prefetch is able to fill in before page faults are happening? Or maybe you are hitting the disk cache?

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by ImprobableTruth on Friday March 31, 2023 @07:55PM (#35395412)

What does it look like if used context size increases?

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by shakow on Friday March 31, 2023 @08:12PM (#35395525)

Disk accesses should not lie. If only 6GiB are read from the disk, then I believe either the model is indeed sparse in its computation, or there may be a bug somewhere.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by outofpaper on Friday March 31, 2023 @08:52PM (#35395843)

You couldn't have said it clearer.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by leereeves on Friday March 31, 2023 @08:16PM (#35395565)

> You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use.

How is that possible? Is the model being compressed even more (even after converting to 4 bit) somehow? Or is most of the model unused?

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by pritambaral on Saturday April 01, 2023 @03:20AM (#35398012)

mmap-ed memory pages backed by a file that aren't dirty aren't counted in an process's RSS usage, only kernel page cache. The mmap-ed regions of virtual memory does get counted in VSZ (virtual memory) but that is just virtual and can be larger than RAM+swap.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by microtherion on Friday March 31, 2023 @09:57PM (#35396256)

> htop still reports only like 4GB of RAM is in use

I think that's just an accounting thing. Many UNIX variants will not "charge" read only memory mapped pages to a process, because they could be shared among many processes and evicted at will.

[ Parent | on HN ]