Re:AMD Discloses Initial Zen 2 Details
by Dylan16807 on Monday November 19, 2018 @12:53AM
(#18484456)
The interconnect latencies give you something like an extra 50 nanoseconds[1]. That matters when accessing memory. Not so much for PCIe, where your base latency is most of a microsecond[2][3]. There seems to be plenty of bandwidth to handle it, too.
[1] https://www.servethehome.com/amd-epyc-infinity-fabric-latenc...
[2] https://forum.stanford.edu/events/posterslides/LowLatencyNet...
[3] https://gianniantichi.github.io/files/papers/pciebench.pdf
My comment was addressing the parent comment about bandwidth, not latency.
Because you only have 1/2 the memory and I/O bandwidth on threadripper, if maxing out I/O is your concern, you would likely be better off with a low-end epyc that still had all the memory controllers and pcie lanes wired up.
Re:AMD Discloses Initial Zen 2 Details
by Dylan16807 on Monday November 19, 2018 @09:10AM
(#18486479)
> My comment was addressing the parent comment about bandwidth, not latency.
Oh, it is? It really seemed to be about direct vs. non-direct, and nothing else. And that pretty much only affects latency, because it only loads the infinity fabric by about 20-25% to route 16 PCIe lanes to each die.
This is part of what the GP was complaining about. The 2 dies which do not have memory also do not have PCIe wired out of the socket, so they are one hop away from I/O as well. If you are trying to max out I/O, you'd probably be better off with a low-end Epyc that had fewer cores enabled per die.
It is interesting to note that the memoryless dies do have PCIe root ports, for built-in things like the PSP crypto engines. That surprised me somewhat when I first noticed it, but it makes sense in retrospect, since the issue is that the TR socket was designed for 2 dies.