Llama.cpp 30B runs with only 6GB of RAM now

Re:Llama.cpp 30B runs with only 6GB of RAM now by bugglebeetle on Monday April 03, 2023 @01:24AM (#35418871)

Yeah, I’ve already been running llama.cpp locally, but not found it to perform at the level attested in the comment (30B model as a chat bot on commodity hardware). 13B runs okay, but inference appears generally too slow on to do anything useful on my MacBook. I wondered what you might be doing to get usable performance in that context.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by MacsHeadroom on Tuesday April 04, 2023 @11:37AM (#35441374)

You can change the number of threads llama.cpp uses with the -t argument. By default it only uses 4. For example, if your CPU has 16 physical cores then you can run ./main -m model.bin -t 16

16 cores would be about 4x faster than the default 4 cores. Eventually you hit memory bottlenecks. So 32 cores is not twice as fast as 13 cores unfortunately.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by bugglebeetle on Tuesday April 04, 2023 @02:19PM (#35443868)

Thanks! Will test that out!

[ Parent | on HN ]