News for hackers. Stuff that matters.

A comment on Turtles on the Wire: Understanding How the OS Uses the Modern NIC (2016). Parent comment

Re:Turtles on the Wire: Understanding How the OS Uses the Mo... by en4bz on Sunday April 02, 2017 @11:35PM (#14020437)
Now that I've reviewed the Chelsio doc more carefully I've noticed that DDP is actually part of iWarp so the TCP offload you mentioned is just one part of iWarp that happens to be transparent to user space which is quite interesting. That being said iWarp has lost out to RoCE mainly because of latency and not throughput. I guess that I'm just more biased towards lower latency (at the same bandwidth) because it is empirically better, even though for most applications 10us vs 1us is negligible. That and because I work in low latency trading.
[ Parent | on HN ]
Re:Turtles on the Wire: Understanding How the OS Uses the Mo... by kev009 on Monday April 03, 2017 @01:15AM (#14020735)
It's kind of the other way around, TCP is done in silicon and firmware and passed up to the OS via a scatter-gather engine which is the core architectural feature. So iWARP lends itself to layering on the TOE, as do other higher level protocols like iSCSI and FCoE. DDP requires intertwining with the OS networking stack and VM but the card is able to DMA the data from the offloaded TCP stream right into the application's address space sockbuf.

I'm out of my league on extreme low latency stuff but take a look at http://www.chelsio.com/wp-content/uploads/resources/T6-WDLat.... For comparison sake, do you know the end to end latency of Mellanox RoCE?

[ Parent | on HN ]
Re:Turtles on the Wire: Understanding How the OS Uses the Mo... by en4bz on Monday April 03, 2017 @02:00AM (#14020876)
Yeah my rather old ConnectX-3 Pro 40Gbe is crushing all these numbers for the given sizes of 16 to 1024.

ib_send_lat is 0.77, 0.77, 0.80, 0.86, 1.17, 1.31, 1.58

ib_write_lat is 0.73, 0.75, 0.76, 0.85, 1.12, 1.29, 1.57

ib_read_lat is 1.38, 1.38, 1.39, 1.42, 1.50, 1.65, 1.91

However it's really difficult to judge these things without a detailed description of the test setup. This is over physical loopback on a single machine and even things like cable length can skew things at this level.

[ Parent | on HN ]