It's little more than an educated guess on my part, but I figure there's about two orders of magnitude in improvements in processing speed exploitable with current processes, if a big-budget chip were designed specifically for ML training. GPUs are architecturally not very optimal for the task.

You want something like a chip with a huge mesh of small independent cores with their own local storage, quite possibly with non-digital circuits that can very quickly approximate the functions with analog electronics, rather than actually doing all of the calculations digitally. Some variation on that is the approach both Intel and IBM have taken with their "neural chips" in the last few years.

It seems that analog computers are finally getting their revenge.

This doesn't seem to match my experience with ML and GPUs/ASICs.

TPU is the main ML ASIC in use. A major goal of the original TPU design seems to be reducing the number of memory accesses. The other top-end ML device is NVIDIA's GPUs with Tensor Cores. Both of those chips are designed around fast matrix multiplication, which right now seems to be the most important operation in deep learning - see how RNNs have started to fall out of favor to CNN-based networks with attention heads.

The TPU is not faster than NVIDIA's GPUs, but it is cheaper. Right now the future seems to be cheaper ML devices designed to be horizontally scalable.

From the CPU perspective, it appears that the major ML effort is related to vectorizing instructions via advanced instruction sets.

Everyone who creates silicon is focused very heavily on using smaller and smaller numeric types - float16 is standard and there is work being done for even smaller int based work.

I haven't seen any analog-based ML devices in use. Can you share an example? Is there even a way to approximate the results of a matmul using analog devices?

It's impossible to guess how much more speed we can get with current approaches, but everything from silicon to networking stack to libraries to network architectures are in their infancy so I would expect dramatic improvements in performance on a regular basis (but not as regular as other areas of software because silicon development is slow)

TPU v3 is rated 420 teraflops, while V100 GPU is rated 125 teraflops.

What does TPU v3 mean there? tpu v3.8? In which case you are comparing 8 cores/4 chips to a single GPU which hardly seems fair. It's hard to compare across ASICs. In practice the the largest readily available amount of compute seem to be 8 V100s vs one 'Cloud TPU' (tpu v3.32). Those two have relatively similar performance in practice (FLOPs seem to be a very poor way to compare across ASICs), although TPU is typically several times less expensive.