News for hackers. Stuff that matters.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by drewg123 on Sunday September 19, 2021 @11:20AM (#28584996)
FreeBSD was selected at the outset of the Open Connect CDN (~2012 or so).

We did a bake off a few years ago, and FOR THIS WORKLOAD FreeBSD outperformed Linux. I don't want to get into an OS war, that's not productive.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by drewg123 on Sunday September 19, 2021 @12:32PM (#28585620)
Its important to consider that we've poured man years into this workload on FreeBSD. Just off the top of my head, we've worked on in house, and/or contributed to or funded, or encouraged vendors to pursue: - async sendfile (so sendfile does not block, and you don't need thread pools or AIO) - RACK and BBR TCP in FreeBSD (for good QoE) - kTLS (so you can keep using sendfile with tls, saves ~60% CPU over reading data into userspace and encrypting there) - Numa - kTLS offload (to save memory bandwidth by moving crypto to the NIC)

Not to mention tons of VM system and scheduler improvements which have been motivated by our workload.

FreeBSD itself has improved tremendously over the last few releases in terms of scalability

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by webmobdev on Monday September 20, 2021 @03:02AM (#28590998)
> FreeBSD itself has improved tremendously over the last few releases in terms of scalability

True. FreeBSD (or its variants) has always been a better performer than Linux in the server segment. Before Linux became popular (mostly due to better hardware support), xBSD servers were famous for their low maintenance and high uptime (and still are). This archived page of NetCraft statistics ( https://web.archive.org/web/20040615000000*/http://uptime.ne... ) provides an interesting glimpse into internet history of how 10+ years back, the top 50 server with the highest uptimes were often xBSD servers, and how Windows and Linux servers slowly replaced xBSD.

(Here's an old HN discussion about a FreeBSD server that ran for 18 years - https://news.ycombinator.com/item?id=10951220 ).

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by sandGorgon on Sunday September 19, 2021 @01:40PM (#28586249)
a lot of this is in Linux now right ? i am asking a personal opinion and not necessarily a "why-dont-u-move-to-linux" question.

Genuinely curious on where u see state of art when it comes to Linux.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by drewg123 on Sunday September 19, 2021 @01:50PM (#28586334)
Yes. I ran it for a bake off ~2 years ago. At the time, the code in linux was pretty raw, and I had to fix a bug in their SW kTLS that caused data corruption that was visible to clients. So I worry that it was not in frequent use at the time, though it may be now.

My understanding is that they don't do 0-copy inline ktls, but I could be wrong about that.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by spockz on Sunday September 19, 2021 @01:40PM (#28586256)
Thank you for pushing kTLS!
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by jenny91 on Sunday September 19, 2021 @12:03PM (#28585394)
Was licensing also a contributing factor, or was that irrelevant for you?
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by drewg123 on Sunday September 19, 2021 @12:17PM (#28585508)
My understanding is that licensing did factor into the decision. However, I didn't join Netflix until after the decision had been made.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by tatersolid on Sunday September 19, 2021 @04:09PM (#28587473)
I recall reading that Netflix chose FreeBSD a decade ago due to asynchronous disk IO was (and still is?) broken and/or limited to fixed block offsets. So nginx just works better on FreeBSD versus Linux for serving static files from spinning rust or SSD.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by Sesse__ on Sunday September 19, 2021 @06:01PM (#28588355)
This used to be the case, but with io_uring, Linux has very much non-broken buffered async I/O. (Windows has copied io_uring pretty much verbatim now, but that's a different story.)
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by gavinray on Sunday September 19, 2021 @06:23PM (#28588498)
Could you expand more on the Windows io_uring bit please?

I have run Debian based Linux my entire life and recently moved circumstantially to Windows. I have no idea how it's kernel model works and I find io_uring exciting.

Wasn't aware of any adoption of io_uring ideas in Windows land, sounds interesting

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by tatersolid on Sunday September 19, 2021 @08:39PM (#28589293)
Windows has had “IO completion ports” since the 1990s which work well and are high performance async for disk/network/other IO operations.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by Sesse__ on Monday September 20, 2021 @03:22AM (#28591086)
This isn't the same as the old Windows async I/O. ptrwis' links are what I thought of (and it's essentially a 1:1 copy of io_uring, as I understand it).
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by ptrwis on Sunday September 19, 2021 @07:27PM (#28588900)
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by kayson on Sunday September 19, 2021 @12:47PM (#28585762)
By how much?