News for hackers. Stuff that matters.

A comment on Llama.cpp 30B runs with only 6GB of RAM now. Parent comment

Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Friday March 31, 2023 @08:46PM (#35395793)
Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who are supposed to be solving them would never even notice; in fact, they'd probably think you're trolling them if you ever brought it up.
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by axlee on Friday March 31, 2023 @09:21PM (#35396016)
Tragedy of the commons only work for things you don't directly pay for.
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by nablags on Friday March 31, 2023 @11:32PM (#35396838)
well in a way - open source software something that you don’t directly pay for
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by sli on Saturday April 01, 2023 @12:28AM (#35397151)
More saliently, the overwhelming majority of the Linux kernel's direct and extended userbase has contributed nothing at all directly to the Linux kernel, as just one example.
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by ElectricalUnion on Saturday April 01, 2023 @12:52AM (#35397255)
Exactly, the software supplier isn't paying for RAM.
[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by somesortofsystm on Saturday April 01, 2023 @07:31AM (#35399394)
>Tragedy of the commons.

Tragedy of folks forgetting how to program.

This mmap() "trick" isn't a trick, its a standard practice for anyone who has cut their teeth on POSIX or embedded. See also mlock()/munlock() ..

[ Parent | on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Saturday April 01, 2023 @08:36AM (#35399782)
Well that's exactly the thing. They haven't. We're talking about a group of people here who live inside scientific papers and jupyter notebooks. They're able to make machines literally think, but you'd be pushing them out of their comfort zone if you stuck them in front of something like Emacs and GCC. Some people like GG, Jeff Dean, etc. are strong in both skill sets, but they're outliers.
[ Parent | on HN ]