I went to a talk from the CTO of Graphcore (https://www.graphcore.ai/) on Monday. They are designing chips targeted at machine learning. As I understood it, their architecture comprises
- lots of "tiles" - small processing cores with collocated memory (essentially DSPs)
- very high bandwidth (90TB/s!) switching fabric to move data between tiles
- "Bulk Synchronous Parallel" operation, meaning that the tiles do their work and then the switching fabric moves the data, and then we repeat.
The key challenge he pointed to was power - both in terms of getting energy in (modern CPUs/GPUs take similar current to your car starter motor!) and also getting the heat out. Logic gates take a lot more power than RAM, so he argued that collocating small chunks of RAM right next to your processing core was much better from a power perspective (meaning you could then pack yet more into your chip) as well as obviously being better from a performance perspective.
https://www.youtube.com/watch?v=Gh-Tff7DdzU isn't quite the presentation I saw, but it has quite a lot of overlap.
Hope that helps!
>"LeCun and other scholars of machine learning know that if you were starting with a blank sheet of paper, an Nvidia GPU would not be the ideal chip to build. Because of the way machine-learning algorithms work, they are bumping up against limitations in the way a GPU is designed. GPUs can actually degrade the machine learning’s neural network, LeCun observed.
“The solution is a different architecture, one more specialized for neural networks,” said LeCun."
Could someone explain to me what exactly are the limitations of current GPGUs such as those sold by Nvidia when used in machine learning/AI contexts? Are these limitation only experienced at scale? Ff someone has resources or links they could share regarding these limitations and better designs I would greatly appreciate it.