I mean most CDN and FANG run on Linux, I think in that case it's kTLS that makes a big difference the rest not much.

Async sendfile is also an advantage for FreeBSD. It is specific to FreeBSD. It allows an nginx worker to send from a file that's cold on disk without blocking, and without resorting to threadpools using a thread-per-file.

The gist is that the sendfile() call stages the pages waiting to be read in the socket buffer, and marks the mbufs with M_NOTREADY (so they cannot be sent by TCP). When the disk read completes, a sendfile callback happens in the context of the disk ithread. This clears the M_NOTREADY flag and tells TCP they are ready to be sent. See https://www.nginx.com/blog/nginx-and-netflix-contribute-new-...

sendfile() with splice and io_uring is similar? I know that this is very experimental on Linux.

The overall idea is to copy bytes from disk to the socket with almost no allocation and not blocking, this is the idea right?

Maybe. A few things in io-uring are implemented by letting a kernel task/thread block on doing the actual work. Which calls that are seems to change in every version, and might be tricky to find out without reading the kernel code.

I'd imagine scaling, licensing and overhead all had something to do with it, too.

Interesting. Are there any benchmarks that you would recommend to look at, regarding FreeBSD vs Linux networking performance?

For anyone interested, here is benchs from late 2018, comparing Fedora and FreeBSD performance: https://matteocroce.medium.com/linux-and-freebsd-networking-...

Why does the author put so much effort into testing VMs? Bare metal installations aren't even tried, so the article won't represent more typical setup (unless you want to run in cloud, in which case it would make sense to test in cloud).

If given the choice I'd never run anything on bare metal again. Let's say we have some service we want to run on a bare metal server. For not very much more hardware money, amortized, I can set up two or three VMs to do the same server, duplicated. Then if any subset of that metal goes bad, a replica/duplicate is already ready to go. There's no network overhead, etc.

I've been doing this for stuff like SMTPE authority servers and ntpd and things that absolutely cannot go down, for over a decade.

That doesn't really matter for benchmarking purposes. I think the parent comment was emphasizing that syscall benchmarks don't make sense when you're running through a hypervisor, since you're running tacitly different instructions than would be run on a bare-metal or provisioned server.

Because the cards were in PCI passthrough, so the performance was exactly the same of a physical system

Author mentions as much as to say that there's was some indirectness at least in the interrupts.

There are also VirtIO drivers involved, and according to the article, they had effect too.

It’s probably worth noting that there have been huge scalability improvements - including introduction of epochs (+/- RCU) - in FreeBSD over the last few years, for both networking and VFS.

Nginx and OpenSSL are open source. Give it a try and reproduce their results with Linux ;-).

IMO the question was reasonable, whereas the answers like yours have always sounded to me like "fuck you."

It was done a tested more than once. As I recall, it took quite a bit to get Linux to perform to the level that BSD was performing for (a) this use can and (b) the years of investment Netflix had already put into the BSD systems.

So, could Linux be tweaked and made as performant for _this_ use case. I expect so. The question to be answered is _why_.

sendfile + kTLS. I'm unaware of the in-kernel TLS implementation for Linux. Is there any around?

Yes, Linux has kTLS. When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists. Mellanox, for example, developed their inline hardware kTLS offload on linux.

> When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists.

Do you have any additional references around this? I'm aware that most rarely used functionality is often broken and therefore usually don't recommend people to use it, but would like to learn about kTLS in particular. I think for Linux OpenSSL 3 now added support for it in userspace. But there's also the kernel components as well as drivers - all of them could have their set of issues.

I recall that simple transmits from offset 0..N in a file worked. But range requests of the form N..N+2MB lead to corrupt data. Its been 2+ years, and I heard it was later fixed in Linux.

I've used sendfile + kTLS on Linux for a similar use case. It worked fine from the start, was broken in two (?) kernel releases for some use cases, and now works fine again from what I can tell. This is software kTLS, though; haven't tried hardware (not the least because it easily saturates 40 Gbit/sec, and I just don't have that level of traffic).

I once recommended a switch and router upgrade to allow for more, new WAPs for an office that was increasingly becoming dependent on laptops and video conferencing. I went with brand new kit, like just released earlier in the year because I'd heard good things about the traffic shaping, etc.

Well, the printers wouldn't pair with the new APs, certain laptops with fruit logos would intermittently drop connection, and so on.

I probably will never use that brand again, even though they escalated and promised patches quickly - within 6 hours they had found the issue and we're working on fixing it, but the damage to my reputation was already done.

Since then I've always demanded to be able to test any new idea/kit/service for at least a week or two just to see if I can break it.

Interesting. I had imagined the range handling is purely handled by the reading side of things, and wouldn't care how the sink is implemented (kTLS, TLS, a pipe, etc). So I assumed the offset should be invisible for kTLS, which just sees a stream of data as usual.

https://www.kernel.org/doc/html/latest/networking/tls-offloa... https://docs.mellanox.com/display/OFEDv521040/Kernel+Transpo...