Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by cokernel_hacker on Tuesday January 04, 2022 @04:30AM
(#29792958)
The conclusions section of the paper is a good summary:
"In the process, we learned ten lessons about DSAs and
DNNs in general and about DNN DSAs specifically that
shaped the design of TPUv4i:
1. Logic improves more quickly than wires and SRAM
⇒ TPUv4i has 4 MXUs per core vs 2 for TPUv3 and 1 for
TPUv1/v2.
2. Leverage existing compiler optimizations
⇒ TPUv4i evolved from TPUv3 instead of being a brand
new ISA.
3. Design for perf/TCO instead of perf/CapEx
⇒ TDP is low, CMEM/HBM are fast, and the die is not big.
4. Backwards ML compatibility enables rapid deployment
of trained DNNs
⇒TPUv4i supports bf16 and avoids arithmetic problems by
looking like TPUv3 from the XLA compiler’s perspective.
5. Inference DSAs need air cooling for global scale
⇒ Its design and 1.0 GHz clock lowers its TDP to 175W.
6. Some inference apps need floating point arithmetic
⇒ It supports bf16 and int8, so quantization is optional.
7. Production inference normally needs multi-tenancy
⇒ TPUv4i’s HBM capacity can support multiple tenants.
8. DNNs grow ~1.5x annually in memory and compute
⇒ To support DNN growth, TPUv4i has 4 MXUs, fast onand off-chip memory, and ICI to link 4 adjacent TPUs.
9. DNN workloads evolve with DNN breakthroughs
⇒ Its programmability and software stack help pace DNNs.
10. The inference SLO is P99 latency, not batch size
⇒ Backwards ML compatible training tailors DNNs to
TPUv4i, yielding batch sizes of 8–128 that raise throughput
and meet SLOs. Applications do not restrict batch size."
Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by ArtWomb on Tuesday January 04, 2022 @09:06AM
(#29794844)
>>> 8. DNNs grow ~1.5x annually in memory and compute
Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests a real breakthrough is needed to keep pace with future requirements. Each MXU already handles 16k ops per tick. And with the additional constraint of optimizing per watt rather than dollar, its quite the challange ;)
Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by omegalulw on Tuesday January 04, 2022 @12:08PM
(#29797061)
> But this suggests a real breakthrough is needed to keep pace with future requirements.
Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.
Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by bsder on Tuesday January 04, 2022 @01:43AM
(#29792086)
Power is a problem and is scaling far less than the node scaling. Raw single threaded performance basically isn't scaling at all.
Going from 45nm to 7nm we only gained about a factor of two in terms of power consumption. Most of that is due to the fact that interconnect and SRAM aren't scaling.
The upshot is that it is worth doing almost infinite amounts of calculation in order to avoid moving anything from point A to point B in a compute unit. If you must use memory, DRAM is death so put big amounts of SRAM on chip and make the buses as wide as possible.
Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by sillysaurusx on Tuesday January 04, 2022 @01:51AM
(#29792124)
Fun fact: this pdf was paywalled, but thanks to some generous members of my Twitter, it was able to be freed. I look forward to a day when all such work is available to anyone who wants it.
Normally I’d have something else to add, but I feel down lately, and it’s nice to see that I’m able to help with one or two things that matter.
I was mostly surprised that the paywall model still persisted in 2021.
Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by sillysaurusx on Tuesday January 04, 2022 @05:11AM
(#29793226)
Yeah, sorry for the confusion. I asked gwern to mirror the liberated pdf, since his /docs/ folder will probably be around a lot longer than me. https://www.gwern.net/docs/index There's a lot of neat stuff in there.
By the way, if any of you happen to run into paywalled pdfs, try posting the paywall link to https://reddit.com/r/scholar. It's surprising how often (and how quickly) that gets you the pdf.
"In the process, we learned ten lessons about DSAs and DNNs in general and about DNN DSAs specifically that shaped the design of TPUv4i:
1. Logic improves more quickly than wires and SRAM ⇒ TPUv4i has 4 MXUs per core vs 2 for TPUv3 and 1 for TPUv1/v2.
2. Leverage existing compiler optimizations ⇒ TPUv4i evolved from TPUv3 instead of being a brand new ISA.
3. Design for perf/TCO instead of perf/CapEx ⇒ TDP is low, CMEM/HBM are fast, and the die is not big.
4. Backwards ML compatibility enables rapid deployment of trained DNNs ⇒TPUv4i supports bf16 and avoids arithmetic problems by looking like TPUv3 from the XLA compiler’s perspective.
5. Inference DSAs need air cooling for global scale ⇒ Its design and 1.0 GHz clock lowers its TDP to 175W.
6. Some inference apps need floating point arithmetic ⇒ It supports bf16 and int8, so quantization is optional.
7. Production inference normally needs multi-tenancy ⇒ TPUv4i’s HBM capacity can support multiple tenants.
8. DNNs grow ~1.5x annually in memory and compute ⇒ To support DNN growth, TPUv4i has 4 MXUs, fast onand off-chip memory, and ICI to link 4 adjacent TPUs.
9. DNN workloads evolve with DNN breakthroughs ⇒ Its programmability and software stack help pace DNNs.
10. The inference SLO is P99 latency, not batch size ⇒ Backwards ML compatible training tailors DNNs to TPUv4i, yielding batch sizes of 8–128 that raise throughput and meet SLOs. Applications do not restrict batch size."
Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests a real breakthrough is needed to keep pace with future requirements. Each MXU already handles 16k ops per tick. And with the additional constraint of optimizing per watt rather than dollar, its quite the challange ;)
Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.
Going from 45nm to 7nm we only gained about a factor of two in terms of power consumption. Most of that is due to the fact that interconnect and SRAM aren't scaling.
The upshot is that it is worth doing almost infinite amounts of calculation in order to avoid moving anything from point A to point B in a compute unit. If you must use memory, DRAM is death so put big amounts of SRAM on chip and make the buses as wide as possible.
Note: I work for Google, but speak for myself.
Normally I’d have something else to add, but I feel down lately, and it’s nice to see that I’m able to help with one or two things that matter.
I was mostly surprised that the paywall model still persisted in 2021.
By the way, if any of you happen to run into paywalled pdfs, try posting the paywall link to https://reddit.com/r/scholar. It's surprising how often (and how quickly) that gets you the pdf.