Re:Exo: Run your own AI cluster at home with everyday devices ⌚
by alexandercheema on Tuesday July 16, 2024 @03:48AM
(#40974403)
Repo author here. That's correct. The embeddings for Llama-3-8B are around 8KB-10KB. For Llama-3-70B they're around 32KB. These are small enough to send around between devices on a local network. For a SETI@home style network, latency will kill you if you go over the internet. That's why we're starting with local networks.
[ Parent
| on HN ]
Re:Exo: Run your own AI cluster at home with everyday devices ⌚
by mg on Tuesday July 16, 2024 @05:12AM
(#40974793)
Ah yes. At first, I thought that since it is all one-way forward-only communication, latency would only affect the time to the first token.
But I guess the final output needs to be sent back to the first node before it can continue. So if there are 50 nodes with a latency of 40ms each, each token would take 2s to process.
Re:Exo: Run your own AI cluster at home with everyday devices ⌚
by alexandercheema on Tuesday July 16, 2024 @05:19AM
(#40974826)
Yeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on throughput with pipeline parallelism, where you overlap execution. See: https://pytorch.org/docs/stable/pipeline.html
Re:Exo: Run your own AI cluster at home with everyday devices ⌚
by juvo on Tuesday July 16, 2024 @06:49AM
(#40975291)
how does it compare to https://github.com/bigscience-workshop/petals ?
Re:Exo: Run your own AI cluster at home with everyday devices ⌚
by DiogoSnows on Tuesday July 16, 2024 @06:09PM
(#40980804)
For generating synthetic data you could have a SETI@Home setup if you consider each home as a node that generates some amount of data.
I mean, such a setup can be built with Exo, I wouldn’t suggest including it as part of Exo.
Out of curiosity, would you ever support training or at least fine-tuning?