The heavy lifting done in neural networks is all offloaded to C++ code and the GPU. Very little of the computation time is spent in python.
ASICs will definitely be very helpful, and there's currently a bit of a rush to develop them. Google's TPUs might be one of the first efforts, but several other companies and startups are looking to have offerings too.
Re:A week-long programming retreat
by davidcuddeback on Monday March 05, 2018 @02:56PM
(#16523848)
> Google's TPUs might be one of the first efforts, but several other companies and startups are looking to have offerings too.
NVidia's Volta series includes tensor cores as well [1]. So far, I think they've only released the datacenter version, which is available on EC2 p3 instances [2].
[1] https://www.nvidia.com/en-us/data-center/volta-gpu-architect...
[2] https://aws.amazon.com/ec2/instance-types/p3/#Product_Detail...
With how fast things are evolving, developing something like an ASIC ($$$) for this might be outdated before it even hits release, no?