SLI is only for realtime rendering with Direct3D and OpenGL. Tesla GPU products are intended primarily for CUDA and OpenCL use, which have always required the application to be aware of the underlying hardware configuration. There's no need to pretend that the 16 GPUs are one super-powerful accelerator device.

They're treated as independent compute units and the host program load-balances compute work between them. You can even do P2P memory copy or access between your CUDA devices without going through the CPU. See https://www.nvidia.com/docs/IO/116711/sc11-multi-gpu.pdf for a high-level overview.

Some of the high end Xeon systems have gobs of PCIe lanes. If you have a four CPU socket system, you can hang four x16 PCIe cards off of each processor.

4CPU sockets !=> 4x x16 cars. It always depends on the number of PCI lanes for each processors. Check i.e. these 2 E7 Xeons: http://www.intel.com/content/www/us/en/products/compare-prod...

Both have max 32 PCI Express lanes - so you can theoretically hang max 2 PCIx x16 cards off each CPU, but as few of the lanes is reserved for other peripherals, in 4GPU/CPU setup they will practically run at lower speeds, ie 8/8/4/4. However, using system with PCIe root complex [1] can improve the GPU<->GPU communication speed.

[1] https://www.microway.com/hpc-tech-tips/pci-express-root-comp...

Or get specialized board with PCIe switch ie. http://www.cirrascale.com/blog/index.php/welcome-big-sur/

so GPUs get better bandwidth (P2P etc).