News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by abhimskywalker on Saturday April 01, 2023 @12:16AM (#35397063)
"The recent change also means you can run multiple LLaMA ./main processes at the same time, and they'll all share the same memory resources." So this could have a main and multiple sub-worker llm processes possibly collaborating while sharing same memory footprint?
Re:Llama.cpp 30B runs with only 6GB of RAM now by l33tman on Saturday April 01, 2023 @04:52AM (#35398555)
Yes, if the model is mmap'ed read-only (as I'm sure it is).

There are other bottlenecks than CPU cores though, it might not be very useful to run multiple in parallel..