News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by sceadu on Friday March 31, 2023 @08:06PM (#35395484)
Maybe we can repurpose the SETI@home infrastructure :)
Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Friday March 31, 2023 @11:26PM (#35396801)
BOINC might be usable but the existing distributed training setups assume all nodes have very high speed I/O so they can trade gradients and model updates around quickly. The kind of setup that's feasible for BOINC is "here's a dataset shard, here's the last epoch, send me back gradients and I'll average them with the other ones I get to make the next epoch". This is quite a bit different from, say, the single-node case which is entirely serial and model updates happen every step rather than epoch.