News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by l33tman on Saturday April 01, 2023 @04:52AM (#35398555)
Yes, if the model is mmap'ed read-only (as I'm sure it is).

There are other bottlenecks than CPU cores though, it might not be very useful to run multiple in parallel..

There are no comments on this comment yet.