Specifically gradient descent is a post hoc approach to network tuning, while human neural connections are reinforced simultaneously as they fire together. The post hoc approach restricts the scope of the latent representations a network learns because such representations must serve a specific purpose (descending the gradient), while the human mind works by generating representations spontaneously at multiple levels of abstraction without any specific or immediate purpose in mind.
I believe the brain's ability to spontaneously generate latent representations capable of interacting with one another in a shared latent space is functionally enabled by the paradigm of neurons 'firing and wiring' together. I also believe it is the brain's ability to spontaneously generate hierarchically abstract representations in a shared space that is the key to AGI. We must therefore move away from gradient descent.
Same thing as humans, just more obvious/visible.
I did the numbers a while ago and honestly I don't think we need smaller transistors to get the computation volume of our mushy brains -- although ofc, more and smaller transistors is always very nice. I believe the only thing stopping AGI at this point is architecture -- we really have no idea how to connect and structure something as complex as our brains -- and cognitive maturity. The last part is my way of saying "two weeks for training a NN? Wait until you have a kid and have to work on training the little human for decades....".
TBH, the ethical implications of AGI seem insurmountable to me. Life is a game --meaning the universe doesn't care about us, nor we owe anything to it--, and for now, it's our game. So, I would rather we put all that computing to improving human life -- including mind upload,-- and put AGI right there with nuclear weapons.
Recently Intel acquired Habana Labs for $2B [1] and Intel could possibly integrate this into upcoming CPUs (Intel sells more CPUs than iPhone for sure). However, this was on the inference side - unlike Cerebras which is making training faster. The most likely products to benefit from this would be Azure or AWS.
1. https://newsroom.intel.com/news-releases/intel-ai-acquisitio...
Amazon/Google/Microsoft will gladly take your money for time on their nVidia GPU instances, but they charge tens of cents per hour.
I tend to be optimistic, so take my prediction with a grain of salt: I bet within 7 or 8 years there will be an inexpensive device that will blow away what we have now. There are so many applications for much larger end to end models that will but pressure on the market for something much better than what we have now. BTW, the ability to efficiently run models on my new iPhone 11 Pro is impressive and I have to wonder if the market for super fast hardware for training models might match the smartphone market. For this to happen, we need a deep learning rules the world shift. BTW, off topic, but I don’t think deep learning gets us to AGI.