News for hackers. Stuff that matters.
Re:GPU-Oriented PCIe Expansion Cluster by krautsourced on Sunday February 26, 2017 @12:27PM (#13738395)
For those doubting the usefulness of these rigs, I can say that for GPU rendering numerous of these are in use by small sized studios and freelancers. The bottleneck of the slow PCIe connection is nothing compared to the render times per frame in most cases. It's only when a frame takes less than a second to render that you notice. They are very popular with e.g. Octane users. The only thing to be aware of is that some motherboards do not play well with this and will not recognize all cards or will not run stable with it attached.
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @10:17AM (#13737562)
The right way to build this sort of thing is with a hub of 8796 PCIE switches for each group of 4 GPUs such that they could all form a continuous 16x PCIE bidirectional ring suitable for O(n) collectives like gather/reduce/allGather/allReduce.

This is more or less useless, or if one wishes to be kind, an amazing emulation of building distributed GPU code over the craptastic bandwidths brought to you by AWS, Microsoft, and Google datacenters for now.

Also PCIE Gen 2? WTF? This might not even post with current GPUs.

Re:GPU-Oriented PCIe Expansion Cluster by chx on Sunday February 26, 2017 @11:04AM (#13737831)
PCIe is all kinds of forgiving. It works all in possible combos.
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @11:10AM (#13737876)
Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so.

Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.

Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @05:56PM (#13740499)
I'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?
Re:GPU-Oriented PCIe Expansion Cluster by Unklejoe on Sunday February 26, 2017 @06:07PM (#13740548)
A somewhat related question:

Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time?

Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!

Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @07:44PM (#13741004)
The TLP header has the destination device's address (in the PCIe tree), so switches just switch. The BARs are related to the host memory map, which has no inherent meaning to devices. nVidia GPU direct introduces a virtual memory region that maps device memory, but I'm not sure if they internally have an IOMMU. For transfers to a device on another bus on the same system the remote device can't be addressed, so it has to ping-pong through system memory, but a device in a different node can be addressed.

The device could query the root complex (the OS) to find the bus address of the device with a particular host memory address but I don't know if that is a common idiom.

(How this works with Multi-Root IO Virtualisation I'm not sure and it gives me a headache.)

Re:GPU-Oriented PCIe Expansion Cluster by chx on Sunday February 26, 2017 @09:01PM (#13741311)
Sure, if you want performance but you said it might not POST. That's not true.
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Monday February 27, 2017 @09:15AM (#13744295)
Yes, it is. For example, in my own experience, Maxwell GPUs frequently do not post with SandyBridge CPUs on some X79 motherboards (Asus P9X79E-WS is one). There are BIOS fixes for this, but many of those motherboards were discontinued before NVIDIA GTX 980 in September of 2014.

Ripping out the SandyBridge CPU and replacing it with an IvyBridge CPU magically fixes the problem. I think the difference here is that SandyBridge only supports PCIE Gen 2 while IvyBridge is PCIE Gen 3. One will never know for sure though.

Re:GPU-Oriented PCIe Expansion Cluster by kobeya on Sunday February 26, 2017 @12:24PM (#13738374)
This is for mining or password cracking.
Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @06:13PM (#13740568)
Both bitcoin and litecoin are both far faster on ASICs. CrytoNight (Monero) is designed to be harder for ASICs but it is early days, I would expect new chips to make a significant improvement over GPUs. Since the memory requirement is only a few MB there is no need for access to large memories or a PCIe bus, and they will continue using ethernet.

https://bytecoin.org/blog/proof-of-work-part-2/

Re:GPU-Oriented PCIe Expansion Cluster by kobeya on Monday February 27, 2017 @03:23AM (#13742767)
There are many others that do not (yet) have ASIC implementations.
Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Monday February 27, 2017 @08:28AM (#13743972)
But does anyone use them? And do they use any schemes that are not (a) a basic hash, (b) some scrypt scheme? Any reason why you wouldn't just put 3 GPUs in a regular chassis and be done for a quarter of the price?
Re:GPU-Oriented PCIe Expansion Cluster by wolrah on Monday February 27, 2017 @11:48AM (#13745540)
> Any reason why you wouldn't just put 3 GPUs in a regular chassis and be done for a quarter of the price?

Not a *coiner but I can guessL

With most motherboards even 3 GPUs will require you to have the cards right up next to each other. This is not great for cooling, which is important when you have the GPUs cranking away at full bore.

Also you can only get 4 GPUs at max in to a normal size PC case. With these you can have a single host PC running 16+ GPUs.

Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @12:41PM (#13738472)
Sure, great, but if they could build this to support Deep Learning and renegade HPC, I suspect they could sell 10,000+ units/year on the down-low. That would be $10M in revenue, maybe twice that if it also worked with Macs.

NVIDIA will do their worst to shut this down because it's a direct threat to DGX-1 running neural networks that aren't entirely communication-limited (long story), but if they could throw this together, I think they could make a great quick buck before the axe falls.

Re:GPU-Oriented PCIe Expansion Cluster by minxomat on Sunday February 26, 2017 @01:17PM (#13738682)
If you take a look at their clients, it's mostly massive companies that probably order large lots of OEM solutions. I don't think amfeltec needs the scale or rather isn't already at the scale you are talking about (margin, not turnover), by enterprise tax and support for OEM builds alone.

The 4x splitters go for about $200. But if you want guaranteed compatibility (i.e. a full build), the price (and margin for them) will skyrocket.

Re:GPU-Oriented PCIe Expansion Cluster by Unklejoe on Sunday February 26, 2017 @06:10PM (#13740562)
How could Nvidia really shut them down? It seems like they're just selling PCIe switches. I could be missing something, but I don't see anything even remotely infringing about this.
Re:GPU-Oriented PCIe Expansion Cluster by jackyinger on Sunday February 26, 2017 @11:18AM (#13737935)
PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...
Re:GPU-Oriented PCIe Expansion Cluster by varelse on Sunday February 26, 2017 @12:33PM (#13738433)
Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so.

Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s).

There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1.

That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim.

And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.

Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @10:08AM (#13737523)
Unfortunately this setup would be very slow for most GPGPU applications, because the CPU-GPU and GPU-GPU bandwidth is very slow. To effectively use this the data transfer requirements have to be very low.

Even with a dedicated 16x PCIe 3 connection there is a latency overhead compared to inter-CPU buses, like HyperTransport or QPI, which is why nVidia and IBM have scaled up the NVlink inter-GPU bus to become a memory speed interconnect.

https://www.ibm.com/blogs/systems/ibm-power8-cpu-and-nvidia-...

Re:GPU-Oriented PCIe Expansion Cluster by EthanV2 on Sunday February 26, 2017 @08:29AM (#13737106)
I'm not as clued up as I used to be about this stuff, but wouldn't this have a pretty serious impact on the performance of the individual cards? Seems like splitting 4 16x cards off one 4x bus would limit the available bandwidth somewhat.
Re:GPU-Oriented PCIe Expansion Cluster by minxomat on Sunday February 26, 2017 @11:47AM (#13738139)
OP here. I found exactly one person using it, with positive results: https://hashcat.net/forum/thread-2622.html

Though amfeltec is sadly a pretty unknown company. Probably best known for their "squid" PCIe "split" cards for multiple M.2 drives in one x16 slot. I've used one of them.

Re:GPU-Oriented PCIe Expansion Cluster by mschuster91 on Sunday February 26, 2017 @08:31AM (#13737111)
That depends if the workload is data-transfer-bound or computing-bound.

If the former, yes you will suffer a massive performance blow even with just one GPU - but if the latter, it's an easy way to upgrade your system.

Re:GPU-Oriented PCIe Expansion Cluster by EthanV2 on Sunday February 26, 2017 @08:34AM (#13737119)
I suppose if you're just working on a data set that's already stored in memory on the GPU(s) the initial work involved in getting that data to the card would be impacted but everything after that benefits from having an absurd amount of computing power
Re:GPU-Oriented PCIe Expansion Cluster by rasz_pl on Sunday February 26, 2017 @09:03AM (#13737232)
This might be useful for password cracking on GPUs. I imagine there is negligent traffic on the bus in this scenario.
Re:GPU-Oriented PCIe Expansion Cluster by mrbill on Sunday February 26, 2017 @09:07AM (#13737256)
I saw a lot of this when GPUs were being used to mine bitcoin/altcoin.
Re:GPU-Oriented PCIe Expansion Cluster by DennisP on Sunday February 26, 2017 @10:19AM (#13737577)
They're still being used for some blockchains with ASIC-resistant mining, including zcash and ethereum.
Re:GPU-Oriented PCIe Expansion Cluster by kierank on Sunday February 26, 2017 @09:20AM (#13737318)
If there are any reasonably-priced ways of doing this especially with more PCIe lanes please let me know. In my experience it's often easier to just buy more motherboards and CPUs than invest in PCIe expansion.
Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Sunday February 26, 2017 @06:44PM (#13740727)
Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures:

https://www.microsemi.com/products/drivers-interfaces-and-pc...

The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html

I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders.

https://events.linuxfoundation.org/sites/events/files/slides...

Re:GPU-Oriented PCIe Expansion Cluster by kierank on Monday February 27, 2017 @06:41PM (#13749687)
The Dolphin systems seem more expensive than just buying 4U servers.
Re:GPU-Oriented PCIe Expansion Cluster by angry_octet on Monday February 27, 2017 @09:03PM (#13750521)
You only buy them when you can't fit your problem into one server (i.e. not enough memory or I/O) or you need redundancy (multipath I/O, synchronised memory). In those cases they can be a lot quicker/cheaper/more reliable than trying to solve it with more computers, protocols and ethernet.
Re:GPU-Oriented PCIe Expansion Cluster by anoother on Sunday February 26, 2017 @11:21AM (#13737956)
A used Tesla s1075 from eBay looks like a good bet. That's my plan, anyway.

Would require some modding to use more up-to-date GPUs (there are some mounting studs on the chassis floor that need trimming) but the forced airflow means that passive cards like the original Xeon Phis are also a possibility.

Re:GPU-Oriented PCIe Expansion Cluster by chx on Sunday February 26, 2017 @11:05AM (#13737836)
This needs a (2014) in the title.
Re:GPU-Oriented PCIe Expansion Cluster by westmeal on Sunday February 26, 2017 @09:28AM (#13737351)
This is interesting but would different architectures of GPUs play nicely? E.g. 3 AMD gpus and one NVIDIA gpu?
Re:GPU-Oriented PCIe Expansion Cluster by Kubuxu on Sunday February 26, 2017 @09:49AM (#13737446)
They should. I have currently AMD + NVIDIA GPU in my PC, the lanes are split on the motherboard (it can be 1 x16 or 2 x8).

There is a bit of fiddling with drivers but it is possible to have one manufacturer working in compute only mode and other as graphics and compute.

Re:GPU-Oriented PCIe Expansion Cluster by throwawayish on Sunday February 26, 2017 @04:36PM (#13740046)
If you think regular GPU drivers are shitty, then wait until you try to install both AMD and nVidia drivers at the same time. It's not very stable, to say the least (Windows 7). Never tried it since. Not worth the hassle.
Re:GPU-Oriented PCIe Expansion Cluster by dragandj on Sunday February 26, 2017 @07:29PM (#13740939)
Easy to install both nvidia and AMD and works like a charm for GPGPU on Arch Linux, though.
Re:GPU-Oriented PCIe Expansion Cluster by minxomat on Sunday February 26, 2017 @11:53AM (#13738165)
Take a look at the hashcat link I posted above. Someone had success with a mixed AMD/NV setup using linux.
Re:GPU-Oriented PCIe Expansion Cluster by StavrosK on Sunday February 26, 2017 @08:49AM (#13737182)
At those speeds, doesn't the cable/bus length play a huge role? What's the maximum length that can support them? I'd imagine it'd be a few centimeters at most...
Re:GPU-Oriented PCIe Expansion Cluster by nabla9 on Sunday February 26, 2017 @09:41AM (#13737405)
Gen 3 PCIe with optical cables works up to 100 meters.
Re:GPU-Oriented PCIe Expansion Cluster by lkjasldkjf on Sunday February 26, 2017 @06:03PM (#13740531)
Seems strange they are using cat 6 cable, which is specified/characterized/tested to 500 MHz, at 2500 MHz for gen 2. Loss is probably not an issue because I see a redriver on the adapter card, but this doesn't seems like a robust solution. I suspect there is a lot of variation in cat 6 cable behavior (across vendors, cable, crimping) at 2.5 GHz.
Re:GPU-Oriented PCIe Expansion Cluster by detaro on Sunday February 26, 2017 @08:56AM (#13737208)
shorter is obviously better, but from what I've read 30 cm PCIe riser cables work, and there are 50 cm examples.
Re:GPU-Oriented PCIe Expansion Cluster by kierank on Sunday February 26, 2017 @09:19AM (#13737312)
We have problems with signal integrity on cheap risers and ribbon cables.
Re:GPU-Oriented PCIe Expansion Cluster by StavrosK on Sunday February 26, 2017 @09:00AM (#13737220)
Huh, that's longer than I thought, thanks.
Re:GPU-Oriented PCIe Expansion Cluster by Radle on Sunday February 26, 2017 @09:12AM (#13737283)
As long as the insulation is good, in theory everything is possible.

You could build your own super long PCI Extender Cable, using superconductors...

Re:GPU-Oriented PCIe Expansion Cluster by throwawayish on Sunday February 26, 2017 @04:33PM (#13740030)
Resistive losses are only a part of the total loss on a transmission line.