News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by l33tman on Saturday April 01, 2023 @04:36AM (#35398468)
Well for starters, web deployment isn't "everything". Python is the de-facto go-to language for research or general prototyping, where not everyone is a programming wiz keeping track of the latest trendy new compiled language. Not everyone can compile stuff even.. :)

Having said that, I've deployed two large Django projects on the web with tons of customers and it runs and scales just fine, and it's a DREAM to maintain and develop for than for example Java.. I would go so far as to say the opposite, if you haven't used Python for web deployment you've been missing out! (you lose some efficiency I'm sure but you gain other things)

[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by pharmakom on Saturday April 01, 2023 @07:01AM (#35399210)
I was talking about running in the Web browser. it's not everything, but it's an important part of everything in my book.
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by sireat on Saturday April 01, 2023 @01:43PM (#35402261)
https://github.com/pyodide/pyodide is pretty amazing for running Python client side in the browser.

You could run notebooks entirely client side https://jupyterlite.readthedocs.io/en/latest/

The startup is slow but otherwise it is pretty functional.

[ Parent | on HN ]