Re:Nvidia will build 700-petaflop supercomputer for Universi...
by xxxtentachyon on Friday July 24, 2020 @09:43AM
(#23939364)
Last time I worked on TPUs, a lot of very pervasive LA operations (e.g. Cholesky decomposition) were unoptimized and slow compared to the NN-style operations. I’m sure that can be fixed, but for the time being, it seems inappropriate for anything beyond the obvious NN operations.
Re:Nvidia will build 700-petaflop supercomputer for Universi...
by rrss on Friday July 24, 2020 @11:59AM
(#23941045)
If "this market" means supercomputers (as 'Cthulhu_ was talking about), then TPUs are certainly not perfectly positioned to capture this market, since they are only capable of doing bfloat16 arithmetic with high performance, and AFAIK incapable of fp64 at all. That means a HPL score of zero.
TPUs will probably dominate AI training, but not so much supercomputing.
Re:Nvidia will build 700-petaflop supercomputer for Universi...
by saberience on Friday July 24, 2020 @07:16AM
(#23938104)
1 milion per second of anything isn't very fast. Unless you're running that on a mobile phone.
GPT-2 117M training at 1 million tokens/sec.
Now, I don't have experience with DGX clusters, so I'm not going to make a firm statement. What I will say is that I, as an outsider, managed to achieve a performance level that is ~unheard of for GPT-2 training. And you can too; TPUs are pervasive.
A TPUv2-512 isn't even as far as the gas pedal goes, either. v3-512 can train all of imagenet to 75.9% accuracy in 4 minutes: https://twitter.com/theshawwn/status/1223395022814339073
v3-1024 can do it in 2 minutes: https://twitter.com/theshawwn/status/1234654848114520065
I once attached a debugger to a training run during startup, after the infeed loop began (meaning it was feeding inputs to the TPU, but no training was happening yet; it was "winding up") and was shocked to discover that when I hit c to continue, it trained on all of imagenet in like 54 seconds. That blows the lid off of every perf result here (under "image classification"): https://mlperf.org/training-results-0-6
(It's not a fair comparison, but it was quite astonishing to see the raw horsepower in action.)
So, nVidia has some catching up to do. And I don't know if they'll be able to. The TPU ecosystem may be clunky at the moment, but boy is it effective. Your options are to invest your time in this ecosystem, which will likely be around in ten years, or in DGX-cluster-type knowledge, which ... might be less pervasive in 10 years.
The distinguishing feature of a TPU is that it has a CPU on board. In fact, it has a CPU with 300GB of memory for every 8 cores. Friggin' love these things.