But why would NVIDIA do that when they can just sell you an A100 for ten times the price of a 4090?

We need AMD to compete, but from what I know their software is subpar to NVIDIA's offering and most of the current ML stacks are built around CUDA. Still there's a lot of money to be made in this area now so competition big and small should pop up.

I'd love it if AMD and Intel teamed up to make a wrapper layer for CUDA. Surely they'd both benefit greatly.

First Intel and then AMD funded a wrapper, yes. Unfortunately the new version supports AMD but no longer Intel.

https://github.com/vosen/ZLUDA

That's a binary level wrapper. Of course there's also ROCm HIP at the source level, and many other things, such as SYCL

In a hypothetical near-future world, competition?

the memory is inherent to the gpu architecture. You cannot just add VRAM and expect no other bottlenecks to pop up. Yes they can reduce the VRAM to create budget models and save a bit here and there. But adding VRAM to a top model is a tricky endeavour

There is a mini-industry in China buying old NVidia GPUs, upgrading the memory, and reselling them.

What's the best that can be achieved with this method?

2X. Converting old NVidia 2080s from 11GB to 22GB. [1]

[1] https://www.tomshardware.com/pc-components/gpus/chinese-work...