At least back in the day, there was talk about Nvidia + ppc64le. Wasn't there even a supercomputer with that setup? But I guess that has fallen by the wayside.

Yes. There was a generation of POWER that offered NVLink.

In 2019 the two fastest machines in the world, Summit & Sierra (see top500.org) used ppc64le + Nvidia GPUs.

We have some older Power9 with NVIDIA V100. Nvidia drivers stopped a while ago (before V100 were considered “old” also for x86_64).

The main problem is software support by most machine learning libraries. While it is normal to have to compile many things from source when the binary is not provided even by some third party, in some cases the software will not compile on ppc64le and surely is not tested to work.

Does anyone else make an ARM CPU that is expressly designed for performance with good cooling ?

It's not entirely clear what you're asking but... AGI, Vera, X2, Spark.

Fujitsu replaced their SPARC CPUs with ARM, I think they were pretty good, but last I read about them was a few years ago.

Fujitsu is developing new ARM based CPUs called Monaka, but their focus seems to be HPC and AI, no mention of "normal" servers

A Fujitsu ARM Server for normal people that I can buy through a normal retail channel and boot a freakin' generic Linux ISO... that would be great!

(Yes, I'm aware of Ampere Altra. The space desperately needs competition and commoditization, imho.)

The funny thing is that Fujitsu has a sizable enterprise IT business, and their Primergy servers are very good, so why don't they offer their own ARM CPUs?

The previous A64fx was just for HPC. Monaka looks more general purpose so maybe they will sell Monaka servers.

A man can dream...

Mainframes are exquisitely "balanced" machines that match processing capacity and IO for a relatively small set of workloads. You won't see mainframes doing scientific number crunching or AI training, but you'll increasingly see them adding more AI inference into their transaction processing workloads for things like fraud and anomaly detection. While their CPUs are prodigiously fast (the one announced is designed to run at 5.7 GHz, and the cores themselves are huge compared to other architectures), a lot of emphasis is placed on the IO systems that connect the machine to storage and other systems so that the CPU is busy all the time. It's very normal to run a mainframe at close to 100% CPU utilization.