if you can fit your model on a single GPU, do that, before parallelizing. Getting more grunt from tightly bound parallelism is one of the hardest problems in software engineering that there is. Weather prediction and climate modelling (two distinct fields in supercomputer) do qualify (you can't fit a high res model of earth in the ram of a single GPU).