the range of technology that needs to come together for ai training is underestimated. there is cuda of course, but there is also nccl, infiniband, gpudirect, each of which requires years of sw and hw maturity. unlike the cpu which has a clean interface (instruction set) the gpu has no such thing - it is more like an octopus with tentacles into networking, compute, storage etc.