IBM has built supercomputers with POWER machines stuffed full of GPUs, in close partnership with Nvidia, for years. I think they might still be the only ones offering systems designed for sustained NVLink bus saturation. So yeah, you could do it - and it would be awesome... but it would be a silly misuse of a mainframe where a cluster would make so much more sense.