Maybe against HN rules, but: thank you. Very helpful comment!
> biological neuron signals aren't quantized
Perhaps, but they inherently have imprecision due the nature of being biological. Expose the same neuron to the same input N times, and you'll get a range of output. The effect of noisy analog data is, broadly speaking, similar to the effect of low-resolution digital data.
Re:How to make LLMs go fast
by throwup238 on Saturday December 23, 2023 @03:34AM
(#38742611)
Expose the same neuron to the same input N times and you won't get a randomly distributed range of outputs, you'll get a signal that is either attenuated or potentiated. That's not noise, that's the simplest form of neuroplasticity at work (which is why static parameters can never represent ion channels).
Beware of feel-good traps like this.
If you were to map the human connectome to a computational neural network down to the ion channel, it'd be at least 500 quadrillion parameters*. That's at least 5-6 orders of magnitude beyond what is currently possible with SOTA ML which means that even if the human brain was 99% devoted to compressing those tokens, that 1% that could actually do work with them is still a thousand times bigger than GPT4. There be emergent dragons.
* This is a fascile argument to begin with since biological neuron signals aren't quantized and ion channels are far too complex to map to a single static parameter