That line refers to training the model from scratch. You can still run the trained model very quickly with one "cheap" GPU.

That said, I'm not sure why one wouldn't get a similar result training on the EC2 or GCE instances that have 8 V100s. Or even training with fewer GPUs but accumulating gradients to get the same batch size.

What software do you use to make those 3D DNN architecture images?

For all the 3D diagrams that I made (including the animated one at the end) I wrote code that used https://threejs.org/ and my custom library. It worked, but with a lot of hassle. In the future I'll likely try using Blender.

I’ve no idea what nvidia use but you could do this pretty easily in Blender with the Freestyle NPR renderer and the built in import images as planes add on.