Re:Llama.cpp 30B runs with only 6GB of RAM now
by sceadu on Friday March 31, 2023 @08:06PM
(#35395484)
Maybe we can repurpose the SETI@home infrastructure :)
Re:Llama.cpp 30B runs with only 6GB of RAM now
by kmeisthax on Friday March 31, 2023 @11:26PM
(#35396801)
BOINC might be usable but the existing distributed training setups assume all nodes have very high speed I/O so they can trade gradients and model updates around quickly. The kind of setup that's feasible for BOINC is "here's a dataset shard, here's the last epoch, send me back gradients and I'll average them with the other ones I get to make the next epoch". This is quite a bit different from, say, the single-node case which is entirely serial and model updates happen every step rather than epoch.
There must be open source projects with enough money to pool into such a project. I wonder whether wikimedia or apache are considering anything.