Re:Andrej Karpathy talks about how Tesla's NNs are structure...
by p1esk on Sunday November 10, 2019 @02:09PM
(#21499560)
I do research in ML hw field: there are currently a couple hundred designs to run a convolutional NN inference. A couple of dozen have been built. They have pretty different underlying technologies (CMOS, floating gate, ReRAM/memristors, etc), different ideas (systolic arrays, analog crossbars, cache organization, lookup tables, data reuse, TDM, using spikes, etc), wildly different power (from microwatts to hundreds of Watts), size, speed, precision, flexibility, cost, ease of use/integration, etc. This is just convnet inference. Lots more is needed to do training in hw, again with multiple choices on how to do it.
So which one design you suggest we all use for all our ML needs?
Re:Andrej Karpathy talks about how Tesla's NNs are structure...
by londons_explore on Monday November 11, 2019 @05:46AM
(#21503695)
The duplication I see in the commercial world in ML inference hardware is in designs similar to the TPU... So a big ~128x28 accumulating mat-mul array, with enough memory throughput to get one operand in and out fast, and enough cache to store the other operand (weights) and switch between which weights are used so the mat-mul array can very efficiently do larger matrix sizes.
Also lookup tables for a bunch of activation functions.
That basic design can efficiently implement nearly any neural net architecture as long as the layer sizes are at least 128x128 and fixed point is okay.
The other exotic designs you suggested are more academic research things, and not yet deployed at scale in anyone's datacenters.
In the self-driving world, the duplication is necessary - different companies are taking different directions, and nobody really knows which will work out.
In the ML hardware world, the duplication is mostly unnecessary. People are developing their own inference hardware ASIC's because they're relatively simple (compared to designing a CPU from scratch, designing a TPU is pretty simple because there are so few operations, and no complex out-of-order execution), and you can't buy one off the shelf yet.
As soon as ML hardware becomes available to buy off the shelf without a massive price premium, everyone will switch to that.