News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by shakow on Friday March 31, 2023 @07:55PM (#35395404)
> Perhaps SWE is dead after all, but LLMs didn't kill it...

Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.

Re:Llama.cpp 30B runs with only 6GB of RAM now by xmprt on Friday March 31, 2023 @08:31PM (#35395692)
This still doesn't make sense. It doesn't take a full year to do optimizations like this. Maybe a month at most if you include the investigation time. And the memory usage is $150 times the number of users which is in the thousands at least.
Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Friday March 31, 2023 @08:46PM (#35395793)
Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who are supposed to be solving them would never even notice; in fact, they'd probably think you're trolling them if you ever brought it up.
Re:Llama.cpp 30B runs with only 6GB of RAM now by axlee on Friday March 31, 2023 @09:21PM (#35396016)
Tragedy of the commons only work for things you don't directly pay for.
Re:Llama.cpp 30B runs with only 6GB of RAM now by nablags on Friday March 31, 2023 @11:32PM (#35396838)
well in a way - open source software something that you don’t directly pay for
Re:Llama.cpp 30B runs with only 6GB of RAM now by sli on Saturday April 01, 2023 @12:28AM (#35397151)
More saliently, the overwhelming majority of the Linux kernel's direct and extended userbase has contributed nothing at all directly to the Linux kernel, as just one example.
Re:Llama.cpp 30B runs with only 6GB of RAM now by ElectricalUnion on Saturday April 01, 2023 @12:52AM (#35397255)
Exactly, the software supplier isn't paying for RAM.
Re:Llama.cpp 30B runs with only 6GB of RAM now by somesortofsystm on Saturday April 01, 2023 @07:31AM (#35399394)
>Tragedy of the commons.

Tragedy of folks forgetting how to program.

This mmap() "trick" isn't a trick, its a standard practice for anyone who has cut their teeth on POSIX or embedded. See also mlock()/munlock() ..

Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Saturday April 01, 2023 @08:36AM (#35399782)
Well that's exactly the thing. They haven't. We're talking about a group of people here who live inside scientific papers and jupyter notebooks. They're able to make machines literally think, but you'd be pushing them out of their comfort zone if you stuck them in front of something like Emacs and GCC. Some people like GG, Jeff Dean, etc. are strong in both skill sets, but they're outliers.
Re:Llama.cpp 30B runs with only 6GB of RAM now by pdntspa on Saturday April 01, 2023 @02:29AM (#35397724)
So let's toss management and go write good code for the principle of it, and not business bullshit calculus
Re:Llama.cpp 30B runs with only 6GB of RAM now by shakow on Saturday April 01, 2023 @10:17AM (#35400498)
Good, tell me how your company will be doing.

What people sometimes fail to understand is that code is a mean to an end, not an end in itself.

If you want to make code for itself, work on an opensource and/or personal project. If you are paid to work on something, you're paid for the something to get out, not for it to feature the best code ever.

Re:Llama.cpp 30B runs with only 6GB of RAM now by pdntspa on Saturday April 01, 2023 @11:11AM (#35400892)
With the margins that tech makes, many companies could certainly afford to care more about code quality. But they don't, instead it gets stuffed into cash reserves where the money sits idle, doing nothing but enriching shareholders.

Or hiring useless business people to install around the periphery of engineering. Which is funny because now tech is letting all those folks go.