Try making a portable package that pulls pytorch with CUDA (reliably) with that set up. I don't even think it's possible as you can't add another package index when using python build.