Re:Llama.cpp 30B runs with only 6GB of RAM now
by chatmasta on Friday March 31, 2023 @07:10PM
(#35395016)
It might not be slow in general, but it's easy to write slow code in it.
[ Parent
| on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now
by Closi on Saturday April 01, 2023 @01:58AM
(#35397547)
In ML python is effectively the interface rather than the bit that is doing the heavy lifting.
The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).