News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by fy20 on Saturday April 01, 2023 @03:13AM (#35397974)
I'd say it's probably not a priority for them right now.

Of course it would save them some money if they could run their models on cheaper hardware, but they've raised $11B so I don't think that's much of a concern right now. Better to spend the efforts on pushing the model forward, which some of these optimisations may make harder.

Re:Llama.cpp 30B runs with only 6GB of RAM now by VernorVintage on Saturday April 01, 2023 @03:55PM (#35403525)
It's a pretty big concern if you had to spend a billion on training, but 6 months later the open source community is able to replicate your training for <100K because you were too cheap to hire an extra 100 optimization experts

That'd be a 10,000 fold depreciation of an asset due to a preventable oversight. Ouchies.