I thought NVIDIA's moat was mostly software/CUDA?

They have by far the fastest chip, the best software with CUDA and feverishly working on next gen chips. They also bought out multiple years worth of global high bandwidth memory manufacturing capacity.

No one will beat them at their game. However if there are any major breakthroughs that might render those processing capacities unneeded, or the major players hitting a wall regarding AI spending, then they will take a massive hit. It will come eventually because the chip business is always in boom/bust cycles.

Yes. Google was building custom HPC hardware 5-8 years before Nvidia decided to expand outside the consumer and "workstation" markets.

Nvidia acquired Mellanox who know far more about custom HPC hardware than Google.

Mellanox and friends not being able to build fast enough switching gear at a reasonable price point was what got Google into the hardware game in the first place ;)