News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by ec109685 on Saturday April 01, 2023 @01:05AM (#35397318)
Before, it had to read the 16GB into a buffer, which itself could be written to disk if the system needs to page out.

That is a big reason startup time is fast.

[ Parent | on HN ]

There are no comments on this comment yet.