I suppose if you're just working on a data set that's already stored in memory on the GPU(s) the initial work involved in getting that data to the card would be impacted but everything after that benefits from having an absurd amount of computing power
If the former, yes you will suffer a massive performance blow even with just one GPU - but if the latter, it's an easy way to upgrade your system.