Llama.cpp 30B runs with only 6GB of RAM now

Re:Llama.cpp 30B runs with only 6GB of RAM now by revelio on Saturday April 01, 2023 @01:57PM (#35402387)

Native library interfacing isn't really Python's strong suit, interpreter plugins are quite painful to write.

.NET has P/Invoke which is much nicer.

JVM is getting Panama+jextract, which is the nicest yet. You can go straight from header files to pure Java bindings which don't need any extra native code at all. But it's not shipped yet :(

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by baq on Saturday April 01, 2023 @03:18PM (#35403179)

Python has had cffi since figuratively forever, so I’m not sure why you compare native modules to P/Invoke?

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by revelio on Sunday April 02, 2023 @06:05AM (#35408954)

Most important libraries with native components are using C extensions, not cffi.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by alanfranz on Saturday April 01, 2023 @08:36PM (#35405781)

What is an “interpreter plugin?” Writing a Python C extension is not that painful, it’s quite well supported. And you’ve got cffi and ctypes as well.

[ Parent | on HN ]