I think a better idea would be an NPU with slower memory, or tie it to the system DDR. I don't think consumer inference (possibly even training) applications would need the memory bandwidth offered by GDDR/HBM. Inference on my 7950x is already stupid fast (all things considered).

The deeper problem is that the market for this is probably incredibly niche.