Re:Llama.cpp 30B runs with only 6GB of RAM now
by startupsfail on Friday March 31, 2023 @07:48PM
(#35395345)
Hmm, can you try running iotop, to see how much is being red from disk? Is it 20Gb or 6Gb? Maybe the prefetch is able to fill in before page faults are happening? Or maybe you are hitting the disk cache?
[ Parent
| on HN ]
There are no comments on this comment yet.