News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by freehorse on Saturday April 01, 2023 @10:42AM (#35400659)
Even with 7.5GB/s you are gonna at best achieve 2.7 seconds for a computing a token, in a hyperoptimistic scenario that you can actually achieve that speed in reading the file, which is too slow for doing much. Maybe if one could get the kernel to swap more aggressively or sth it could cut half that time or so, but it still would be quite slow.

There are no comments on this comment yet.