Re:Llama.cpp 30B runs with only 6GB of RAM now
by diimdeep on Saturday April 01, 2023 @07:21AM
(#35399319)
Also, current SSD's achieve 7.5 GB/s+ read speed, opposed to older SSD from 2013 with 500 MB/s, so performance will drastically differ depending on your system specs in case of pulling weights from disk to RAM on demand.
Also, there is $ vmmap <pid> where we can see various statistics about process memory and used swap, that are not available in top or htop.
[ Parent
| on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now
by freehorse on Saturday April 01, 2023 @10:42AM
(#35400659)
Even with 7.5GB/s you are gonna at best achieve 2.7 seconds for a computing a token, in a hyperoptimistic scenario that you can actually achieve that speed in reading the file, which is too slow for doing much. Maybe if one could get the kernel to swap more aggressively or sth it could cut half that time or so, but it still would be quite slow.