News for hackers. Stuff that matters.

A comment on Exo: Run your own AI cluster at home with everyday devices ⌚. Parent comment

Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by mg on Tuesday July 16, 2024 @03:03AM (#40974203)
Interesting. Let's do the math ...

Let's say the model has 50B parameters and 50 layers. That would mean about one billion values have to travel through the wifi for every generated token?

I wonder how much data that is in bytes and how long it takes to transfer them.

[ Parent | on HN ]
Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by blackbear_ on Tuesday July 16, 2024 @03:28AM (#40974324)
It's not the parameters that are sent, it's the layer outputs. That makes for a few thousands floats per token
[ Parent | on HN ]
Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by mg on Tuesday July 16, 2024 @03:37AM (#40974355)
Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are probably right and the number of parameters is much higher.

A few thousand floats would be just around 10K bytes, right? Those can be sent in a fraction of a second. Consumer internet speed is at what ... 10MB/s? So we could build a SETI@home style network of layers and produce 1K tokens per second?

[ Parent | on HN ]
Re:Exo: Run your own AI cluster at home with everyday devices ⌚ by tama_sala on Tuesday July 16, 2024 @01:46PM (#40978645)
The embedding size is only 8k so while the parameters are 70B. So it's a huge difference
[ Parent | on HN ]