It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale analogue circuits to operate even when 8-bit accuracy).
Very interesting chips.
It sorta makes sense, considering your body is generally electrically conductive everywhere. Signals propagate as waves of charge sustained by cells opening and closing little pores that selectively release ions. "Conductors" have pores, and "insulators" don't. That's also why nerve signals are relatively slow to propagate vs. straight up electricity. That's also why you can measure muscle activity via an ECG; you're measuring the transient voltages being generated by the waves of charge moving around. The voltages themselves don't have much meaning to your body. Of course, if you apply enough voltage to build up a charge inside a muscle, you can cause it to actuate.
I recently had to get a pacemaker, and apparently the particular lower threshold to trigger my heart muscle to contract is around 0.7V for 350uS, at whatever impedance the lead happens to be. Below that, and my heart muscle does nothing. Anything above that, any my heart muscle does a full beat. The device applies a 2V pulse so that there's plenty of margin, and can go up to 5V if needed, in case the lead's impedance increases. The lower the voltage the better, in terms of battery life. The cool thing is that the device can safely coexist with any natural electrical activity my faulty nerves may have, since the muscle simply responds to whichever pulse happens to arrive first.
The human brain would like to have a word with you.
[0] https://en.wikipedia.org/wiki/Refractory_period_%28physiolog...
You should note the O(log n) delay growth is for a logic (ideal) model. Considering the 2D geometry of real circuits, there are models with more detail, in particular VLSI models. [1]
I suspect due to routing and path lengths even the delay would turn out to be O(n) in VLSI, or at least a power of n (with a small constant).
[1] Seems like a good introduction: http://cs.brown.edu/people/jsavage/book/pdfs/ModelsOfComputa...
Someday I'll sit down and actually try to design a prototype for such a thing.
No, it's O(1). The bit size of how you represent a single datum does not grow with size of your model.
When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse).
Once you learn how to slow down the earlier bits, you end up with all the bits arriving at the same time, and you can have up to n adds pipelined and timed to the clock when output matters.
But with NN I think now would be a fun time (and likely the last decade as well) to rethink the basics right down to the basics. Everything need not be a NAND gate. :)