Turtles on the Wire: Understanding How the OS Uses the Modern NIC (2016)

Re:Turtles on the Wire: Understanding How the OS Uses the Mo... by kev009 on Monday April 03, 2017 @01:15AM (#14020735)

It's kind of the other way around, TCP is done in silicon and firmware and passed up to the OS via a scatter-gather engine which is the core architectural feature. So iWARP lends itself to layering on the TOE, as do other higher level protocols like iSCSI and FCoE. DDP requires intertwining with the OS networking stack and VM but the card is able to DMA the data from the offloaded TCP stream right into the application's address space sockbuf.

I'm out of my league on extreme low latency stuff but take a look at http://www.chelsio.com/wp-content/uploads/resources/T6-WDLat.... For comparison sake, do you know the end to end latency of Mellanox RoCE?

[ Parent | on HN ]

Re:Turtles on the Wire: Understanding How the OS Uses the Mo... by en4bz on Monday April 03, 2017 @02:00AM (#14020876)

Yeah my rather old ConnectX-3 Pro 40Gbe is crushing all these numbers for the given sizes of 16 to 1024.

ib_send_lat is 0.77, 0.77, 0.80, 0.86, 1.17, 1.31, 1.58

ib_write_lat is 0.73, 0.75, 0.76, 0.85, 1.12, 1.29, 1.57

ib_read_lat is 1.38, 1.38, 1.39, 1.42, 1.50, 1.65, 1.91

However it's really difficult to judge these things without a detailed description of the test setup. This is over physical loopback on a single machine and even things like cable length can skew things at this level.

[ Parent | on HN ]