Do you have any numbers you can share publicly? What's the reason it's better (Linux has in-kernel TLS as well, correct?)?. It would make for a great topic for Netflix TechBlog.

Comparing to Linux? No. There are no public numbers.

However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.

Have you guys done a ton of your own customization and modification to the FreeBSD stack?

Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.

Most of our customizations and modifications have been upstreamed. We get similar performance on production 100g and 200g boxes when running the upstream kernel. I see no reason I shouldn't be able to hit 380Gb/s on a single-socket Rome box with an upstream kernel. I just haven't tried yet.

Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.

You can search FreeBSD and Netflix on HN. They have presented their finding over the years with detailed comments in those threads.

Is that with CX7 and Dual Socket AMD CPU you mentioned awhile ago? ( I remember it was lost in delivery or some supply chain issues )

Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.

4x CX6-DX in a dual-socket Rome.

Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.

> At this point, we’re able to serve 100% TLS traffic comfortably at 90 Gbps using the default FreeBSD TCP stack.

https://netflixtechblog.com/serving-100-gbps-from-an-open-co...

That was 2017 -- it's quite a bit higher now.