News for hackers. Stuff that matters.
Re:Pingora, the proxy that connects Cloudflare to the Internet by BonoboIO on Wednesday September 14, 2022 @01:19PM (#32840262)
Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.
Re:Pingora, the proxy that connects Cloudflare to the Internet by totallyunknown on Wednesday September 14, 2022 @04:49PM (#32843162)
A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready.

I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s.

Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.

Re:Pingora, the proxy that connects Cloudflare to the Internet by BonoboIO on Wednesday September 14, 2022 @05:15PM (#32843447)
Es liegt mir auf der Zunge welche Firma das ist.

I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong.

For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.

Re:Pingora, the proxy that connects Cloudflare to the Internet by totallyunknown on Thursday September 15, 2022 @03:33AM (#32848418)
A single node is single socket AMD EPCY node with 32 cores, 64 or 128GB RAM, 1x NIC with 2x100 GB/s. Ja, es ist waipu.tv.
Re:Pingora, the proxy that connects Cloudflare to the Internet by ddorian43 on Thursday September 15, 2022 @03:42AM (#32848517)
What about the disks?

Edit: or linear TV means the recent stream can be all in ram?

Re:Pingora, the proxy that connects Cloudflare to the Internet by totallyunknown on Thursday September 15, 2022 @03:57AM (#32848660)
All in memory.
Re:Pingora, the proxy that connects Cloudflare to the Internet by otbutz on Friday September 16, 2022 @07:41AM (#32864776)
There is quic-go[1] but i don't think that it's sufficiently optimized[2-4] to be used for this kind of workload. caddy will use it to provide HTTP/3 by default[5] in the upcoming 2.6.0 release.

[1]: https://github.com/lucas-clemente/quic-go [2]: https://github.com/lucas-clemente/quic-go/issues/2877 [3]: https://github.com/lucas-clemente/quic-go/issues/2607 [4]: https://github.com/lucas-clemente/quic-go/issues/341 [5]: https://github.com/caddyserver/caddy/pull/4707

Re:Pingora, the proxy that connects Cloudflare to the Internet by lossolo on Wednesday September 14, 2022 @06:56PM (#32844584)
Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ?

1. https://man7.org/linux/man-pages/man2/sendfile.2.html

Re:Pingora, the proxy that connects Cloudflare to the Internet by totallyunknown on Thursday September 15, 2022 @03:35AM (#32848439)
We can't use sendfile since all the traffic is HTTPS. Sendfile can only be used if we can use KTLS. Internally we have some kind of cache hierarchy for video objects, these files are transferred out via Sendfile, but the performance gain here is negligible.
Re:Pingora, the proxy that connects Cloudflare to the Internet by costela on Thursday September 15, 2022 @03:48AM (#32848578)
and yes: the proxy does have some intelligence around "just" forwarding requests. Mostly caching related, but also some upstream routing decisions and a ton of observability.
Re:Pingora, the proxy that connects Cloudflare to the Internet by trillic on Wednesday September 14, 2022 @01:29PM (#32840407)
100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
Re:Pingora, the proxy that connects Cloudflare to the Internet by zamadatix on Thursday September 15, 2022 @09:47PM (#32861031)
40 Mbps seems roughly 5x-10x the bitrate I'd expect for this kind of thing.
Re:Pingora, the proxy that connects Cloudflare to the Internet by totallyunknown on Wednesday September 14, 2022 @04:53PM (#32843205)
100 Gbit/s / 5000 KiB/s is 20000.
Re:Pingora, the proxy that connects Cloudflare to the Internet by ImprobableTruth on Wednesday September 14, 2022 @05:06PM (#32843357)
KiB = Kibibyte, not Kibibit
Re:Pingora, the proxy that connects Cloudflare to the Internet by mplewis on Wednesday September 14, 2022 @03:51PM (#32842458)
“Only” :)
Re:Pingora, the proxy that connects Cloudflare to the Internet by 988747 on Friday September 16, 2022 @08:02AM (#32864942)
Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
Re:Pingora, the proxy that connects Cloudflare to the Internet by nly on Friday September 16, 2022 @08:10AM (#32865025)
It's almost as if on-demand video streaming isnt suitable for broadcasting live events.