I mean a single V100 GPU (~= 100 TFLOPS) has FLOPS throughputs similar to top end mid-00's supercomputers like Blue Gene, at least superficially. And you can squeeze like 4+ in a 1U if you have enough cooling and power.

https://en.wikipedia.org/wiki/History_of_supercomputing