It also depends on the interconnect speed. If you don't have fast enough interconnect between the machines, you won't get linear speedup with N gpus.