Interesting. Let's do the math ...

Let's say the model has 50B parameters and 50 layers. That would mean about one billion values have to travel through the wifi for every generated token?

I wonder how much data that is in bytes and how long it takes to transfer them.

It's not the parameters that are sent, it's the layer outputs. That makes for a few thousands floats per token

Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are probably right and the number of parameters is much higher.

A few thousand floats would be just around 10K bytes, right? Those can be sent in a fraction of a second. Consumer internet speed is at what ... 10MB/s? So we could build a SETI@home style network of layers and produce 1K tokens per second?

The embedding size is only 8k so while the parameters are 70B. So it's a huge difference