News for hackers. Stuff that matters.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by otterley on Sunday September 19, 2021 @11:23AM (#28585019)
How would you characterize the economics of this vs. alternative solutions? Achieving 400Gb/s is certainly a remarkable achievement, but is it the lowest price-per-Gb/s solution compared to alternatives? (Even multiple servers.)
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by drewg123 on Sunday September 19, 2021 @11:27AM (#28585062)
I'm not on the hardware team, so I don't have the cost breakdown. But my understanding is that flash storage is the most expensive line item, and it matters little if its consolidated into one box, or spread over a rack, you still have to pay for it. By serving more from fewer boxes, you can reduce component duplication (cases, mobos, ram, PSUs) and more importantly, power & cooling required.

The real risk is that we introduce a huge blast radius if one of these machines goes down.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by throwdbaaway on Sunday September 19, 2021 @07:07PM (#28588796)
I almost fell for the hype of pcie gen4 after reading https://news.ycombinator.com/item?id=25956670, and it is quite interesting that pcie gen3 nvme drives can still do the job here. What would be the worst case disk I/O throughput while serving 400Gb/s?
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by toast0 on Sunday September 19, 2021 @10:09PM (#28589767)
If you look at just the pci-e lanes and ignore everything else, the NICs are x16 (gen4) and there's two of them. The NVMes are x4 (gen3) and there are 18 of them. Since gen4 is about twice the bandwidth of gen3, it's 32 lanes of gen4 NIC vs about 36 lanes of gen4 equivalent NVMe.

If we're only worried about throughput, and everything works out with queueing, there's no need for gen4 NVMes because the storage has more bandwidth than the network. That doesn't mean gen4 is only hype; if my math is right, you need gen4x16 to have enough bandwidth to run a dual 100G ethernet at line rate, and you could use fewer gen4 storage devices if reducing device count were useful. I think for Netflix, they'd like more storage, so given the storage that fits in their systems, there's no need for gen4 storage; gen4 would probably make sense for their 800Gbps prototype though.

In terms of disk I/O, either in the thread or the slides, drewg123 mentioned only about 10% of requests were served from page cache, leaving 90% served from disk, so that would make worst case look something like 45GB/sec (switching to bytes cause that's how storage throughput is usually measured). From previous discussions and presentations, Netflix doesn't do bulk cache updates during peak times, so they won't have a lot of reads at the same time as a lot of writes.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by throwdbaaway on Sunday September 19, 2021 @11:31PM (#28590152)
Thanks for the numbers. Perhaps hype is not the right word. It is just interesting to see that some older hardware can still be used to achieve the state of the art performance, as the bottleneck may lie elsewhere.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by toast0 on Sunday September 19, 2021 @11:47PM (#28590210)
It's always balancing bottlenecks. Here, the bottleneck is memory bandwidth, limiting to (more or less) 32 lanes of network; the platform has 128 lanes, so using more lanes than needed at a slower rate works and saves a bit of cost (probably). On their Intel Ice Lake test machine, that only had 64 lanes which is also a bottleneck, so they used Gen4 NVMe to get the needed storage bandwidth into the lanes available.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by kofejnik on Sunday September 19, 2021 @07:52PM (#28589062)
thank you for this link, superuseful for my next project
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by sroussey on Sunday September 19, 2021 @02:05PM (#28586479)
Ouch. I guess there is always a trade off to be made…
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by toast0 on Sunday September 19, 2021 @07:29PM (#28588912)
One of the factors is ISPs that install these boxes have limited space and many companies that want to place hardware there. If Netflix can push more traffic in less space, that makes it a better option for ISPs that want to reduce external traffic and thus more likely to be installed benefiting their users.

If what I found is current, Netflix has 1U and 2U appliances, but FB does a 1U switch + groups of 4x 2U servers, so starting at 9U; but I was looking at a 2016 FB pdf that someone uploaded, they may have changed their deployments since then. 2U vs 9U can make a big difference.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by dragontamer on Sunday September 19, 2021 @01:13PM (#28586008)
I doubt that the 32-core EPYC they focused on is even the most economical solution in this situation.

If they're really RAM-bandwidth constrained, then the 24-core 74F3 (which still has all 256MBs of L3 cache) or even 8-core 72F3 may be better.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by sroussey on Sunday September 19, 2021 @02:09PM (#28586513)
L3 won’t matter.

The more compute clusters the more PCI lanes in EPYC and the SSD lanes go direct per 8cores.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by samstave on Sunday September 19, 2021 @05:42PM (#28588213)
I recall when I was at Intel in the 90s and 3MB of L3 was a big deal.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by mcdevilkiller on Monday September 20, 2021 @04:04AM (#28591340)
The 8 core still has 128 lanes and 8 channels of memory, though.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by fragmede on Sunday September 19, 2021 @06:11PM (#28588420)
Particularly, kTLS is a solution that fits here, on a single-box, but I wonder how things would look if the high-perf storage boxes sent video unencrypted and there was a second box that dealt only with the TLS. We'd have to know how many streams 400Gb/s represents though, and have a far more detailed picture of Netflix's TLS needs/usage patterns.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by toast0 on Monday September 20, 2021 @01:45AM (#28590669)
A proxy for TLS wouldn't help with this load.

That proxy would still need to do kTLS to reduce the required memory bandwidth to something the system can manage, and then you're at roughly the same place. The storage nodes would likely still have kTLS capable NICs because those are good 2x100G NICs anyway. It would be easier to NUMA align the load though, so there might be some benefit there. With the right control software, the proxy could pick an origin connection that was NUMA aligned with the client connection on the proxy and the storage on the origin. That's almost definitely not worth doubling the node count for though, even if proxy nodes don't need storage so they're probably significantly less expensive than flash storage nodes.

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by miohtama on Sunday September 19, 2021 @07:01PM (#28588752)
Could Netflix replace TLS with some in-house alternative that would push more processing to the client? Something that pre-encrypts the content on disk before sending, eliminating some of the TLS processing requirements?
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by floatboth on Sunday September 19, 2021 @08:23PM (#28589221)
If you pre-encrypt contents on disk, it wouldn't be a per-user unique key.
Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by fred256 on Sunday September 19, 2021 @10:23PM (#28589835)
The content is already encrypted anyway for DRM.

I'd assume TLS is used in large part for privacy reasons (so ISPs can't sell info on what shows are popular)

Re:Serving Netflix Video at 400Gb/s on FreeBSD [pdf] by pastrami_panda on Sunday September 19, 2021 @11:24AM (#28585038)
This, can someone add to this it's crucial.