I only know of Nervana's ASIC that is promising to be used for training and Google hinted at it for future work. Does anyone know of other processor, ASIC or FPGA's that are built for training?
GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you?
Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling.
As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.
As to whether the TPU can perform efficient training or whether it chokes like current FPGAs due to insufficient memory bandwidth is a different question. That would be my guess but I'd love to be proven wrong.