News for hackers. Stuff that matters.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @10:32AM (#32521448)
Author here. AMA
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by sophacles on Friday August 19, 2022 @11:08AM (#32521946)
Every iteration of this prezzo I've seen over the years has made for a fascinating morning read, thanks!

As much as I enjoy the results of the work, I'm always a bit curious how the sausage is made. Is pushing the hardware limits your primary job or something you do periodically? How do you go about selecting the gear you use? How much do you work with the vendors? (etc etc) I'd really enjoy a behind the scenes blog post or something wrt this serving absurd amounts of traffic from a single box.

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @12:33PM (#32523146)
My role is to make our CDN servers more efficient. One of the easiest and most fun ways to do that is to push servers as hard as I can and see what breaks and what doesn't scale. I also work with our hardware team and their vendors to evaluate new hardware and how it can fit into our system.

But I do plenty of other things as well, including fixing random kernel bugs. You can read the git log of the FreeBSD main branch to see some of the things I've been working on..

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by pyrolistical on Friday August 19, 2022 @01:00PM (#32523509)
When are you going to cut the CPU/main memory out completely?

The bottleneck is at your NIC anyways, so seems like there would be a market for NIC that can directly read from disk into NIC's working memory

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @01:59PM (#32524195)
We've looked at this. The problem is that NICs want to read in TCP MSS size chunks (1448 bytes, for example), while storage devices are highly optimized for block-aligned (4K) chunks. So you need to buffer the storage reads someplace, and for now the only practical answer is host memory. There are NVME technologies that could help, but they are either too small, or come at too large of a price premium. CXL memory looks promising, but its not ready yet.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by Matthias247 on Friday August 19, 2022 @03:09PM (#32524910)
Does it? I thought with segmentation offloads the NIC basically gets TCP stream data in more or less arbitrary sizes, and then segments in into MTU sizes on its own?
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @03:15PM (#32524965)
We do fairly sophisticated TCP pacing, which requires sending down some small multiple of MSS to the NIC, so it doesn't always have the freedom to pull 4K at a time.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by nh2 on Friday August 19, 2022 @03:07PM (#32524889)
You mention AIO in nginx.

In 2021 somebody submitted a patch for io_uring support in nginx:

https://mailman.nginx.org/pipermail/nginx-devel/2021-Februar...

I'm not sure if there has been further progress on it so far. In one comment feedback is "it doesn't seem to make the typical nginx use case much faster" [at that time].

But I find this interesting, because io_uring can make almost all things async that can't be used async so far in Linux (open(), stat(), etc) and thus in nginx.

Would io_uring integration in nginx be relevant for you?

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by w10-1 on Friday August 19, 2022 @01:29PM (#32523841)
Thank you very much "drewg123"!

Future technology advances increasingly looks like this complex work integrating hardware, OS fixes, team collaboration. People and teams and companies working together, and contributing to shared resources like FreeBSD. Tolerating mistakes at scale, giving credit where credit is due, and all the other things that make respect real, which creates a space to get things done.

Most of us will never get close to these opportunities or contexts, but still it helps us advance our own technique/culture to observe and model your story. And perhaps you'll help new collaborators find you. All the best.

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by bri3d on Friday August 19, 2022 @11:46AM (#32522486)
At what point does it make sense to replace the CPU and OS with custom hardware and software? At this point the CPU is basically doing TCP state maintenance and DMA supervision, but not much else, right?

I totally get the cost, convenience, and supply chain risk-value in commodity stuff that you can just go out and buy, but once you're bound to a single network card, this advantage starts to go away, and it seems like you're fighting with the entire system topology when it comes to NUMA, no? Why not a "TCP file send accelerator" instead of a whole computer?

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by wmf on Friday August 19, 2022 @06:16PM (#32526633)
I suppose you could attach NVMe drives directly to Bluefield and cut out x86.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by sam0x17 on Friday August 19, 2022 @01:03PM (#32523545)
> Sendfile

Ah, so this is why everything stutters / falls apart when you switch subtitles on or off -- it has to access a whole different file and resume at the same place in that file I assume? I would think you would want the (verbal) audio separated out in a different file so it can be swapped out on the fly without re-initializing the video stream, and same thing with subtitle files? I'm just making some assumptions based on the behavior I've seen but would be cool to know how this works.

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @03:18PM (#32524996)
No, video and subtitles are separate files.

I've never seen this bad behavior myself. Do you mind sharing the client you're using?

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by sam0x17 on Saturday August 20, 2022 @12:36AM (#32529117)
4k Apple TV
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by throw0101c on Friday August 19, 2022 @10:45AM (#32521636)
You're using ConnectX-6 Dx here. Any technical reason for that particular NIC, or just haven't gotten around to ConnectX-7s yet?

Have you examined other NIC vendors? (Chelsio?)

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @10:54AM (#32521756)
This talk was given roughly 4 months ago. CX7 was not available yet. I'm looking forward to testing on them when we get some.

We looked at Chelsio (as T6 was available well before CX6-DX). However, the CX6-DX offers a killer feature not available on T6. The CX6-DX can remember the crypto state of any in-order stream, while the T6 cannot. That means that the TCP stack can send, say, 4K of a TLS record, wait for acks, and come back 40ms later and send the next 4K and DMA just the requested 4K from the host. The T6 cannot remember the state, and would need to DMA the first 4K (which was already sent) in order to re-establish the crypto state, and then DMA the requested 4K. This could run the PCIe bus out of bandwidth. The alternative is to make TCP always chunk sends at the TLS record size, but this was horrible for streaming quality.

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by phantomathkg on Friday August 19, 2022 @11:43AM (#32522442)
> Serve only static media files

This part I don't get. How about DRM? Unless Netflix pre-DRM all contents for all user?

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by bri3d on Friday August 19, 2022 @12:39PM (#32523230)
Encrypting assets on the fly using a per-consumer symmetric key would be prohibitively expensive, so I'm sure the media is stored pre-encrypted using a shared symmetric key.

It only really matters that this key is unique per package, not per user, because once even a single user can compromise the trusted execution environment and extract either the key or the plain video stream, that piece of content is now pirated anyway. So, key reuse against the same content probably isn't really a major part of the threat model - this attacker could share the key with others, but they might as well share the decrypted content instead.

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @12:29PM (#32523089)
Yes, all our content is also DRMed. Else somebody could easily pirate content..
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by onedr0p on Friday August 19, 2022 @06:38PM (#32526879)
To be fair, it already seems easily pirated. DRM is useless, if content is able to be viewed on some personal device it can be ripped and shared. I'd be curious how much effort/money companies dump into adding DRM measures, it seems like a lost cause. Maybe it just makes the execs sleep better at night.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by Mindwipe on Saturday August 20, 2022 @09:14PM (#32536560)
This is a myth.

Netflix's DRM is sufficiently good that the Reddit Piracy subreddit has spent the last three months moaning that they have no access to 4K Netflix rips, at least for weeks or months after the content comes out.

Netflix's DRM and key management systems do what they care about pretty well at this point, which is protect the initial airing of popular shows.

Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by onedr0p on Friday September 02, 2022 @06:39PM (#32696501)
1080p rips are found within hours of the content first airing on NetFlix. You don't need to download Linux ISOs in full quality to enjoy them.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by Bluecobra on Friday August 19, 2022 @12:24PM (#32523012)
I would think that media files would be already encrypted and gets decrypted by the Netflix client. Otherwise the DRM could easily be defeated by using something like Wireshark.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by ManWith2Plans on Friday August 19, 2022 @12:44PM (#32523297)
Do you have a link to video or audio for this presentation? I'm probably don't speak for just myself when I say I would love to see it.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by quux on Friday August 19, 2022 @01:29PM (#32523836)
Someone else linked the video here: https://news.ycombinator.com/item?id=32520750
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by Bluecobra on Friday August 19, 2022 @10:39AM (#32521555)
What kind of tuning is done in the BIOS? Is that profile available to view to everyone? Are you using a custom BIOS from Dell?
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @10:57AM (#32521815)
Not much tuning needed to be done. The little that was is mentioned in the talk, and was basically to set NPS=1 and to disable DLWM, in order to be able to access the full xgmi interconnect bandwidth at all times, even when the CPU is not heavily loaded.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by gopaz on Friday August 19, 2022 @12:38PM (#32523226)
What about the storage? Is it using Raid? Does blocksize matter? What filsystem is used?
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by drewg123 on Friday August 19, 2022 @01:21PM (#32523755)
Every storage device is independent (no RAID), and runs UFS. We use UFS because, unlike ZFS, it integrates directly with the kernel page cache.
Re:Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf] by ComputerGuru on Friday August 19, 2022 @10:50PM (#32528553)
Which is part of the reason sendfile doesn’t work well for ZFS.