Re:System design implications of declining network latency v...
by jo909 on Monday November 02, 2015 @10:56AM
(#10492520)
Very true. Most servers are multiple processors (sockets) interconnected via a high speed network (QPI, HT), and each processor has links to its own memory and some PCI lanes. You could almost see that as multiple distinct computers on one board and in one case.
Hard disks with direct ethernet connections exist (f.e. Seagate Kintec).
Re:System design implications of declining network latency v...
by pjc50 on Monday November 02, 2015 @11:18AM
(#10492697)
Seagate Kinetic looks like an excellent idea, although horribly expensive.
Re:System design implications of declining network latency v...
by MichaelGG on Monday November 02, 2015 @02:26PM
(#10494421)
It goes deeper than that. Some of those links are called QPI. Accessing memory on a multi-socket system can be much slower (40%) if the memory isn't connected to the socket you're on. In that case the request has to go to the other socket, then to RAM. In a quad-socket system, this might mean making two hops, first.
Even accessing cache in a multi-socket system sucks. Taking ownership of a cache line that's in another socket can cost 300 cycles. Ouch.
So in a way, programming current Intel NUMA systems is a lot more like writing distributed systems in the first place, if you care about performance. Yes, it's more reliable (your cross-socket read isn't gonna sporadically fail), but if you treat it as a single unit, things get slow.
Re:System design implications of declining network latency v...
by acveilleux on Monday November 02, 2015 @09:59AM
(#10492078)
To be fair, PCIe/SATA are a hell of a lot more reliable as "network links" as 10gb ethernet. SANs however, have the same failure modes as typical LANs.
Some of those network links are labelled "PCIe" or "SATA", and some are labelled "Ethernet" or "DisplayPort", but they're all high-speed serial links. A PCIe lane is only 8Gbit, comparable to 10Gbit ethernet.
This means that your computer "backplane" design can be de-constrained; you don't have to draw a hard distinction between elements in the same case and those in the case of the machine next to you in the rack.
A rotating disk is roughly able to saturate a 1Gb ethernet link. Maybe we should just go full SAN and put Ethernet connectors on the drives.