Indeed. But the frustrating thing remains. Obviously animal and human brains do something similar, and obviously it's not even N^2, because anyone can count inputs and outputs for our brains, calculate N^2 and even at picowatts per operation that is just not happening ...
1000 Hz, ~800 input nerves, 20W power usage => 800 000 input values per second, squared, would leave 0.03 nW per input value. That's not enough to turn the handle on a single protein, so that's not happening. That's not enough to propagate the result of the calculation within a cell. A 2.3 power is out of the question.
One imagines maybe you can get to N^1.4 within that budget for whatever the brain is doing. Maybe. But that would be the limit.
Not for matrix multiplication, since the output, which has to be written, is already of size O(n^2).
big constants are not new. We all had a laugh when matrix mul exponent went from [2.3728]639 to [2.3728]596
Even more than that - reading GMP manual and seeing all the "more efficient, but bigger constant" algorithms take over each other as the size grows cause *awe*
so this "sudden" discovery that theory is different from practice has been bloody obvious for everyone. I'm more inclined to say "if you want to went frustration at Ai - do that, stop showing your ignorance"
---
the main benefit for these results isn't to show what's possible - it's to show what's *not impossible*
---
also I just saw that this is the same author that wrote "Did I read this paper? Of course not. No one at OpenAI read this paper either, so why should I?"
so he expects same level of discussion he deserves himself - completely negligible