News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by politician on Friday March 31, 2023 @05:25PM (#35393885)
Roughly: OpenAIs don’t employ enough jarts.

In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas.

When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.

Re:Llama.cpp 30B runs with only 6GB of RAM now by bee_rider on Friday March 31, 2023 @05:32PM (#35393957)
The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.
Re:Llama.cpp 30B runs with only 6GB of RAM now by fallous on Friday March 31, 2023 @07:47PM (#35395341)
The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.
Re:Llama.cpp 30B runs with only 6GB of RAM now by hedgehog on Friday March 31, 2023 @06:00PM (#35394249)
They have very good people but those people have other priorities.
Re:Llama.cpp 30B runs with only 6GB of RAM now by dsjoerg on Friday March 31, 2023 @07:04PM (#35394957)
Re:Llama.cpp 30B runs with only 6GB of RAM now by sp332 on Friday March 31, 2023 @07:18PM (#35395087)
In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America

https://en.m.wikipedia.org/wiki/Justine_Tunney

Re:Llama.cpp 30B runs with only 6GB of RAM now by hackernewds on Friday March 31, 2023 @09:44PM (#35396194)
what's your point? also interestingly JART is in the thread here, so they might have read your comment :)
Re:Llama.cpp 30B runs with only 6GB of RAM now by sp332 on Saturday April 01, 2023 @12:59PM (#35401864)
I'm pretty sure she knows already?
Re:Llama.cpp 30B runs with only 6GB of RAM now by dflock on Friday March 31, 2023 @10:39PM (#35396490)
The jart: the standard unit of developer.

"I've got 20 yrs experience, and I think I'm about 150 milli jarts, maybe 200 on a good day."

Re:Llama.cpp 30B runs with only 6GB of RAM now by fy20 on Saturday April 01, 2023 @03:13AM (#35397974)
I'd say it's probably not a priority for them right now.

Of course it would save them some money if they could run their models on cheaper hardware, but they've raised $11B so I don't think that's much of a concern right now. Better to spend the efforts on pushing the model forward, which some of these optimisations may make harder.

Re:Llama.cpp 30B runs with only 6GB of RAM now by VernorVintage on Saturday April 01, 2023 @03:55PM (#35403525)
It's a pretty big concern if you had to spend a billion on training, but 6 months later the open source community is able to replicate your training for <100K because you were too cheap to hire an extra 100 optimization experts

That'd be a 10,000 fold depreciation of an asset due to a preventable oversight. Ouchies.