News for hackers. Stuff that matters.

A comment on Exo: Run your own AI cluster at home with everyday devices ⌚. Parent comment

Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by blackbear_ on Tuesday July 16, 2024 @03:28AM (#40974324)
It's not the parameters that are sent, it's the layer outputs. That makes for a few thousands floats per token
[ Parent | on HN ]
Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by mg on Tuesday July 16, 2024 @03:37AM (#40974355)
Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are probably right and the number of parameters is much higher.

A few thousand floats would be just around 10K bytes, right? Those can be sent in a fraction of a second. Consumer internet speed is at what ... 10MB/s? So we could build a SETI@home style network of layers and produce 1K tokens per second?

[ Parent | on HN ]
Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by tama_sala on Tuesday July 16, 2024 @01:46PM (#40978645)
The embedding size is only 8k so while the parameters are 70B. So it's a huge difference
[ Parent | on HN ]