News for hackers. Stuff that matters.
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @10:17AM (#13737562)
The right way to build this sort of thing is with a hub of 8796 PCIE switches for each group of 4 GPUs such that they could all form a continuous 16x PCIE bidirectional ring suitable for O(n) collectives like gather/reduce/allGather/allReduce.

This is more or less useless, or if one wishes to be kind, an amazing emulation of building distributed GPU code over the craptastic bandwidths brought to you by AWS, Microsoft, and Google datacenters for now.

Also PCIE Gen 2? WTF? This might not even post with current GPUs.

Re:GPU-Oriented PCIe Expansion Cluster by chx on Sunday February 26, 2017 @11:04AM (#13737831)
PCIe is all kinds of forgiving. It works all in possible combos.
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @11:10AM (#13737876)
Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so.

Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.

Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @05:56PM (#13740499)
I'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?
Re:GPU-Oriented PCIe Expansion Cluster by Unklejoe on Sunday February 26, 2017 @06:07PM (#13740548)
A somewhat related question:

Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time?

Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!

Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @07:44PM (#13741004)
The TLP header has the destination device's address (in the PCIe tree), so switches just switch. The BARs are related to the host memory map, which has no inherent meaning to devices. nVidia GPU direct introduces a virtual memory region that maps device memory, but I'm not sure if they internally have an IOMMU. For transfers to a device on another bus on the same system the remote device can't be addressed, so it has to ping-pong through system memory, but a device in a different node can be addressed.

The device could query the root complex (the OS) to find the bus address of the device with a particular host memory address but I don't know if that is a common idiom.

(How this works with Multi-Root IO Virtualisation I'm not sure and it gives me a headache.)

Re:GPU-Oriented PCIe Expansion Cluster by chx on Sunday February 26, 2017 @09:01PM (#13741311)
Sure, if you want performance but you said it might not POST. That's not true.
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Monday February 27, 2017 @09:15AM (#13744295)
Yes, it is. For example, in my own experience, Maxwell GPUs frequently do not post with SandyBridge CPUs on some X79 motherboards (Asus P9X79E-WS is one). There are BIOS fixes for this, but many of those motherboards were discontinued before NVIDIA GTX 980 in September of 2014.

Ripping out the SandyBridge CPU and replacing it with an IvyBridge CPU magically fixes the problem. I think the difference here is that SandyBridge only supports PCIE Gen 2 while IvyBridge is PCIE Gen 3. One will never know for sure though.

Re:GPU-Oriented PCIe Expansion Cluster by kobeya on Sunday February 26, 2017 @12:24PM (#13738374)
This is for mining or password cracking.
Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @06:13PM (#13740568)
Both bitcoin and litecoin are both far faster on ASICs. CrytoNight (Monero) is designed to be harder for ASICs but it is early days, I would expect new chips to make a significant improvement over GPUs. Since the memory requirement is only a few MB there is no need for access to large memories or a PCIe bus, and they will continue using ethernet.

https://bytecoin.org/blog/proof-of-work-part-2/

Re:GPU-Oriented PCIe Expansion Cluster by kobeya on Monday February 27, 2017 @03:23AM (#13742767)
There are many others that do not (yet) have ASIC implementations.
Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Monday February 27, 2017 @08:28AM (#13743972)
But does anyone use them? And do they use any schemes that are not (a) a basic hash, (b) some scrypt scheme? Any reason why you wouldn't just put 3 GPUs in a regular chassis and be done for a quarter of the price?
Re:GPU-Oriented PCIe Expansion Cluster by wolrah on Monday February 27, 2017 @11:48AM (#13745540)
> Any reason why you wouldn't just put 3 GPUs in a regular chassis and be done for a quarter of the price?

Not a *coiner but I can guessL

With most motherboards even 3 GPUs will require you to have the cards right up next to each other. This is not great for cooling, which is important when you have the GPUs cranking away at full bore.

Also you can only get 4 GPUs at max in to a normal size PC case. With these you can have a single host PC running 16+ GPUs.

Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @12:41PM (#13738472)
Sure, great, but if they could build this to support Deep Learning and renegade HPC, I suspect they could sell 10,000+ units/year on the down-low. That would be $10M in revenue, maybe twice that if it also worked with Macs.

NVIDIA will do their worst to shut this down because it's a direct threat to DGX-1 running neural networks that aren't entirely communication-limited (long story), but if they could throw this together, I think they could make a great quick buck before the axe falls.

Re:GPU-Oriented PCIe Expansion Cluster by minxomat on Sunday February 26, 2017 @01:17PM (#13738682)
If you take a look at their clients, it's mostly massive companies that probably order large lots of OEM solutions. I don't think amfeltec needs the scale or rather isn't already at the scale you are talking about (margin, not turnover), by enterprise tax and support for OEM builds alone.

The 4x splitters go for about $200. But if you want guaranteed compatibility (i.e. a full build), the price (and margin for them) will skyrocket.

Re:GPU-Oriented PCIe Expansion Cluster by Unklejoe on Sunday February 26, 2017 @06:10PM (#13740562)
How could Nvidia really shut them down? It seems like they're just selling PCIe switches. I could be missing something, but I don't see anything even remotely infringing about this.
Re:GPU-Oriented PCIe Expansion Cluster by jackyinger on Sunday February 26, 2017 @11:18AM (#13737935)
PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @12:33PM (#13738433)
Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so.

Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s).

There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1.

That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim.

And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.