Re:Andrej Karpathy talks about how Tesla's NNs are structure...
by londons_explore on Sunday November 10, 2019 @08:44AM
(#21497920)
Training is currently done in floating point math, whereas inference can be done fixed point without much loss of performance. Fixed point is ~10x cheaper in terms of power and silicon area for equal performance.
Also, training requires a lot more RAM per unit of compute, since it needs to store all past layer activations, whereas for inference, that is unnecessary.
As far as I know, no player who has developed dedicated ML hardware (as opposed to using GPU's) uses the same hardware for both inference and training.
Re:Andrej Karpathy talks about how Tesla's NNs are structure...
by modeless on Sunday November 10, 2019 @06:10AM
(#21497513)
Edge TPUs support 8 bit integer math only. Training is floating point. That's not a small change. https://coral.withgoogle.com/docs/edgetpu/models-intro/#mode...
In other words, Cloud TPUs could be the same architecture than Edge TPUs but scaled to an higher frequency and more packed.
I guess we need sources to confirm.