DPDK Lib is good when you want to have user space app working with the IO bypassing the kernel. however it may greatly harm the system performance in the cases when kernel thread will hook the CPU resources and throw away you user space thread. This can introduce milliseconds of latency in your routing path. Also one question do you think to support L3 PFC support ? This is very important in the infrastructures where number of nodes > 1K.

Right. We instruct Kernel to reserve 8 CPU cores exclusively for our user space application that is responsible for traffic forwarding.

Here's performance results. Max latency is 0.31 msec.

####################################################

min_latency_ns: 708

avg_latency_ns: 63723

max_latency_ns: 310795

jitter_ns: 30940

####################################################

Regarding L3 PFC support. In my understanding it's a ROCE requirement. I view RoCE as east-west traffic, since its not leaving the boundaries of the switch fabric, I think non-blocking leaf/spine fabric should be sufficient there. Please correct me if'm missing something.

SoftGate, the Linux router where we use DPDK, is designed for North/South traffic, sitting on the border of the data center, so we don't expect RoCE traffic there, but again, please let me know if you see it differently.

You have got a very good latency numbers. It's impressive.

You are right about RoCE traffic. It is mainly East-West traffic. However NTT has a project called IOWN and in the scope of that project as I know they are trying to build North-South traffic based on the RoCE technology. In the case when you will decide to use your SoftGate routers inside the datacenter it would make sense to have a multipath routing capabilities and also solution for "PFC spreading blocking problem". This problem is not particularly bound with RoCE transport layer but instead for a MPI use cases. A several big infrastructure companies built their own network transport layer and in house hardware to solve those problems.

Can you use it for deep packet inspection process?

You can hook a DPI into Netris operated network, yes.