News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Miraste on Friday March 31, 2023 @06:57PM (#35394897)
Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
Re:Llama.cpp 30B runs with only 6GB of RAM now by chatmasta on Friday March 31, 2023 @07:10PM (#35395016)
It might not be slow in general, but it's easy to write slow code in it.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Closi on Saturday April 01, 2023 @01:58AM (#35397547)
In ML python is effectively the interface rather than the bit that is doing the heavy lifting.

The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).

Re:Llama.cpp 30B runs with only 6GB of RAM now by chaorace on Friday March 31, 2023 @07:54PM (#35395401)
Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Max-Limelihood on Monday April 03, 2023 @06:57PM (#35432517)
In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Max-Limelihood on Tuesday April 04, 2023 @12:42PM (#35442394)
It really is that slow.

You're completely correct that the speed-sensitive parts are written in lower-level libraries, but another way to phrase that is "Python can go really fast, as long as you don't use Python." But this also means ML is effectively hamstrung into only using methods that already exist and have been coded in C++, since anything in Python would be too slow to compete.

There's lots of languages that make good tradeoffs between performance and usability. Python is not one of those languages. It is, at best, only slightly harder to use than Julia, yet orders-of-magnitude slower.