Intel tried that with Intel Xeon Phi, basically.

That was just a lot of Intel cores. I'm talking about an NVIDIA GPU that happens to have some ARM cores along with the CUDA cores.

You're essentially talking about a Jetson:

https://elinux.org/Jetson_AGX_Xavier

Eight Nvidia ARMv8 cores, 512 Volta cores, 64 tensor cores, and 2 DLAs - all connected to 32 GB of shared RAM.

It was announced over two years ago and the price has fallen to $800. For another $200 you can add a USB-C to DP cable (for a second display) and 1TB of NVMe storage.

If I had a $1,000 compute budget it's what I'd get. You'd have a more than functional dual 4K daily driver for dev work, browsing, etc plus the ability to run ML workloads. Oh yeah, and it tops out at less than 50 watts from the wall with all but the heaviest tasks coming in at 25-50% of that.

I love the Jetson -- we have a bunch of them here -- but that's a SBC. What I want is an NVIDIA 2080Ti or 3080Ti card with some ARM cores that I can run apps that use CUDA+ARM directly on!

I love my Jetsons too!

One has to wonder if at some point Nvidia will produce a PCIe carrier board or something for souped up Jetson modules or some variant. I have no idea how the connectivity (PCIe device?) would work out but it's definitely within the realm of possibility.

The Xavier generation supports behaving as a PCIe device (PCIe device mode), so it's maybe not too outlandish. :-)

At that point how is the "GPU" card not just a small motherboard/SoM? I mean, I guess having it run off of a host system still helps with density, but it's 80% of a full computer already at that point.