rwitten

For int8 we've done very precise ablations with deterministic data loading to show incremental impact: https://cloud.google.com/blog/products/compute/accurate-quan...

Totally agreed with all your comments! MaxText mainline is right now a reference implementation for users who have their own scientific opinions on model architecture and convergence. We're additionally hoping to let MaxText run compatibly with some open-source models for customers who want to use known good configurations.

This is a great series of questions and it isn't our goal to prove you otherwise!

We work with customers interested in training models who run their own ablations, including batch size and learning rates.

Based on that, we demonstrate workloads that we think will be interesting to potential customers! Absolutely agreed that this workload has a larger batch size than the public literature suggests.

Great questions! Slices are a set of TPU chips that share a fast, private inter-chip-interconnect. Unlike the current GPU generation in clouds, the TPUs on different machines can communicate through this private network. Multislice means that we're using a hierarchical network, where there is both inter-chip-interconnect and normal data-center netowrking.

More details: https://cloud.google.com/tpu/docs/multislice-introduction

(P.S. - contributor on blog post, Google employee, all thoughts my own)

(Contributor on the blog post, all opinions my own)

Agreed with you and we definitely weren't trying to brag! This is fast compared to people's expectations in the space but slow compared to what we should be able to accomplish and will accomplish in the future.

Hey! I'm an contributor on this (Rafi Witten), all opinions my own.

You're asking the right question but I think the math is off by a bit. The equivalent number on the H100's is 989 TFLOP/s/chip so the equivalent job is ~10K H100's = (10 * 10^18) / (989 * 10^12). (Both chips also have 8-bit acceleration!)

I believe this is the largest ML job both by exaflops and number of chips every demonstrated. Other companies own more chips or exaflops than we show in this job but getting all the hardware working at once on a single job is a different matter! :-)

If you're interested in lower bounds or tighter upper bounds, you can find the latest here:

http://link.springer.com/article/10.1007%2Fs00453-014-9891-7... http://statweb.stanford.edu/~candes/papers/RandomizedNLA.pdf

Yep - an Android app is in the works (probably 3-4 months out). We're happy to ship to England.

Miles and I are on HN or you can shoot us an email at [email protected] - one of us will get to it there.