News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by seydor on Friday March 31, 2023 @06:18PM (#35394466)
> we don't really have a way for the FOSS community to pool together that much money

There must be open source projects with enough money to pool into such a project. I wonder whether wikimedia or apache are considering anything.

Re:Llama.cpp 30B runs with only 6GB of RAM now by sceadu on Friday March 31, 2023 @08:06PM (#35395484)
Maybe we can repurpose the SETI@home infrastructure :)
Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Friday March 31, 2023 @11:26PM (#35396801)
BOINC might be usable but the existing distributed training setups assume all nodes have very high speed I/O so they can trade gradients and model updates around quickly. The kind of setup that's feasible for BOINC is "here's a dataset shard, here's the last epoch, send me back gradients and I'll average them with the other ones I get to make the next epoch". This is quite a bit different from, say, the single-node case which is entirely serial and model updates happen every step rather than epoch.
Re:Llama.cpp 30B runs with only 6GB of RAM now by totony on Saturday April 01, 2023 @02:03AM (#35397575)
Or big cloud platform could give some compute for free, give back some of the profit they get from oss.