Google claims[0] the TPU is many times faster for the workloads they've designed it for.

> On our production AI workloads that utilize neural network inference, the TPU is 15x to 30x faster than contemporary GPUs and CPUs.

As far as I know this will be the first opportunity for the public to prove those claims, as until now they've not been available on GCP. I don't mean to sound skeptical–I'm quite confident they're not exaggerating.

[0]: https://cloudplatform.googleblog.com/2017/04/quantifying-the...

Keep in mind that what you linked refers to TPUv1, which is built for quantized 8-bit inference. The TPUv2, which was announced in this blog post, is for general purpose training and uses 32-bit weights, activations, and gradients.

It will have very different performance characteristics.

Thanks for pointing that out!

The reserve TPU button has been available on the dashboard for the last few months. But I assume instances have been prioritized for large customers such as Two Sigma.

From the paper:

"Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU."

In-Datacenter Performance Analysis of a Tensor Processing Unit

https://arxiv.org/abs/1704.04760

Price is about 5x cloud nvidia gpu instance on an hourly basis.

It will be interesting to see some benchmarks that compare TPUs to V100, since all previously published comparisons from Google compare TPU to K80 (3 GPU architectures ago).

Perf per watt matters to Google but not you. You should only think of it on a perf/$ basis, right?

They're closely related though, since if the perf per watt is lower then Google can charge you less doller per perf. The price they charge you is ultimately tied to the operating cost.

I wonder how these would compare with Amazon's FPGA instances with a comparable core running.

I would imagine that (by design) they're not directly comparable.

I suspect that we'll see more information about the ASICs over time, but it'll take time to really understand their characteristics vs a Nvidia GPU - which are at least right now a bit better understood.

It is. TPUs perform calculations on weights using low-precision floating point and integer types. This saves a ton of computation, but doesn't matter much for training models.

But GPU is also able to use lower resolution types. There must be more to the TPU advantage.

GPUs are much more complex (general-purpose) and therefore cannot be optimized beyond a certain point due to timing requirements and PVT (process, temperature, voltage) variations. In other words, the more stuff you have on an ASIC, the more careful you have to be ensure a margin of tolerance for variations.

So the only advantage of the TPU is it's a simpler and more specialized asic? Google didn't break any new ground in terms of training perf?

So, a way to think of this is: The speed (and therefore, cost) of training a machine learning model depends on (a) the ML techniques (how rapidly the model converges and to what accuracy); and (b) how quickly the processor executes the operations involved in the ML techniques.

The TPU is only an improvement in (b). It's not going to result in a big-O style speedup, because the same training algorithms and architectures will run on it that we run on CPUs & GPUs today.

I'm not sure what counts as "breaking new ground" - is that 10%? 100%? 1000? :-) The things to watch out for in benchmarks will be:

(a) Perf/$. This is actually a big deal - one of my students recently blew through $5000 of Google Cloud credits running Imagenet experiments, in a week. And we didn't finish them! As this cost really drops, it enables things like Neural Architecture Search, which uses tons of compute capability to explore architectural variants automatically.

(b) Absolute perf.

(c) Performance scaling. To what degree will the fast, 2D torroidal mesh allow a full pod of Cloud TPUs to scale nearly-linearly? Absolute training times matter from a user productivity standpoint. Waiting 30 minutes for a result is very different from waiting 12 hours (you can do one of these while you sneak out to go running! :-).

The NIPS'17 slides have more technical context for some of this: https://supercomputersfordl2017.github.io/Presentations/Imag...

> So the only advantage of the TPU is it's a simpler and more specialized asic

And everything that entails: lower energy consumption, higher throughput, lower cost at volume, higher profits for GCP, etc.

> Google didn't break any new ground in terms of training perf?

Relative to GPUs, sure, but I can't say how well they stack up against other custom ASICs for DL applications.

This is a new thing. Google also has Nvidia GPUs. these are new custom designed ASICs google has designed for certain ML tasks.