I think a lot of next generation chips are not just squeezing more compute into hardware, but examining whether we can change the nature of neural net compute ro better fit what we can do in hardware. That's Graphcore's pitch at least. And plenty of people are looking into whether you can do non-IEEE754 math in a way that works for compute, e.g. FP16, quantized arithmetic, etc.

You probably don't need the entire team to be familiar with the math, but someone has to be.

And as other comments have noted, Software is actually going to be a very big deal.