Nvidia tensor cores are small systolic arrays. They'd have to throw out a lot of their ecosystem investments and backwards compatibility to make effective use of them as the main GPU compute, and there's really no need given how competitive their chips are right now.

  Less programmable, but more throughput/power efficiency?

I also wonder the same. It'd make sense to sell two categories of chips:

Traditional GPUs like Blackwell that can do anything and have backwards compatibility.

Less programmable and more ASIC-like inference chips like Google's TPUs. Inference market is going to be multiple times bigger than training soon.

The CUDA ecosystem is their moat.