News for hackers. Stuff that matters.
Re:Why do we use the Linux kernel's TCP stack? (2016) by brendangregg on Saturday October 03, 2020 @03:40PM (#24674191)
The Linux network stack has been rapidly evolving; nowadays we have XDP for a fast path through the kernel. But there's also a lot of moving parts that ensure high throughput on noisy networks, for different application workload patterns, to avoid bufferbloat, and to do this fairly across multiple processes/containers. I'd imagine one can develop a user-space stack that appeared better in a simple iperf microbenchmark in a lab, but would lose for real-world applications and networks (unless your real-world use case was very narrow).

I'd recommend understanding the stack before choosing to replace it (the old programmer's advice about understanding a gate before removing it). I summarized the TCP send path in my next book (sysperf2, with help from Linux network engineers); it involves:

  - Socket send buffers
  - Pacing
  - TSO
  - Congestion controls
  - Nagle
  - TSQ
  - qdisc (optional)
  - GSQ
  - BQL
  - TSO (NIC)
That's just the TCP send path.

I'm happy to expand all these acronyms, but if any are new to you, that's kind-of my point. These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve.

Re:Why do we use the Linux kernel's TCP stack? (2016) by techman9 on Saturday October 03, 2020 @05:13PM (#24674719)
This is super useful! I've been hacking recently on qdisc classifiers, and trying to set a filter up with BPF, but there's a paucity of documentation on the subject. The tc tools are sorta documented here (https://man7.org/linux/man-pages/man8/tc-bpf.8.html), but it's quite hard to find docs on the underlying syscalls and the behavior of `bpf_prog_type_sched_cls`.

I bought a copy of BPF performance tools to learn more about the BPF interface, and it's really useful but quite focused on performance! I wish there was a resource of similar depth and breath about eBPF for classification/XDP.

Re:Why do we use the Linux kernel's TCP stack? (2016) by brendangregg on Saturday October 03, 2020 @07:07PM (#24675338)
Thanks for getting the BPF book. Further references for BPF bits would be the kernel docs (Documentation/bpf), kernel source (kernel/bpf, include/.../bpf), the self tests (tools/testing/selftests/bpf), Cilium's docs (https://docs.cilium.io/en/stable/bpf/), and if all else fails, mailing lists like iovisor-dev.

It sounds like there should be a deep book on BPF internals (written by maintainers Alexei & Daniel), but it would be a tough sell: you really do need the maintainers to author it or be heavily involved to do it well, and it's a lot of work for a changing target, and without a large audience (lots of people will use BPF, but few will code it at that level. I'd guess 100 world-wide. Not a great case for a book!) Compare it to my BPF perf tools book: big audience (end users: developers, SREs, perf engineers, etc), and has a long shelf life (the tools are done, and work, and should keep working).

Re:Why do we use the Linux kernel's TCP stack? (2016) by dilyevsky on Saturday October 03, 2020 @03:48PM (#24674236)
Big fan of your books and blog! Would you say usefulness of dpdk and xdp has diminished purely from performance standpoint? I hear rumors you can do line rate on modern linux kernels these days but haven’t had a chance to try it personally yet. I think at the time maglev was originally developed google was still on 2.6 kernels...
Re:Why do we use the Linux kernel's TCP stack? (2016) by brendangregg on Saturday October 03, 2020 @05:13PM (#24674717)
Thanks. I'd expect more future uses of xdp over dpdk. As for max performance (I saw a Linux server max out multiple 10 GbE ports years ago) I'm not sure anyone right now knows what it is. We've recently added XDP, io_uring, BPF congestion control algorithms, and more. There are performance numbers for these in isolation, but I don't think anyone has performance tested (and tuned) them all working together yet.

And that poses a problem: to reach maximum performance requires many engineering hours to learn and configure all of these. You'd need a large gain to justify doing this: E.g., for a system doing over 1M packets/s.

This complexity may change. Look at BPF for observability: we built BCC and bpftrace so that using it was trivial. What trivial front-end/technology could we build upon XDP and io_uring to make deployment easy?

Re:Why do we use the Linux kernel's TCP stack? (2016) by maartenh on Sunday October 04, 2020 @06:37AM (#24677855)
How would you debug programs using ioring? With the traditional I/O syscalls, one can use tools like strace to observe the behavior of a program.

Would using BPF to instrument the ioring be the way to go?

Re:Why do we use the Linux kernel's TCP stack? (2016) by bogomipz on Saturday October 03, 2020 @06:12PM (#24675064)
>"I'd expect more future uses of xdp over dpdk."

Can say why that is? I haven't really kept up with DPDK but I know there was lots of interest in it a few years back. What has changed?

Re:Why do we use the Linux kernel's TCP stack? (2016) by wtallis on Sunday October 04, 2020 @01:30AM (#24676716)
I haven't been watching the networking side too closely, but I expect the same thing to play out for storage with io_uring vs SPDK. When the in-kernel solution is almost as fast, has much broader availability and compatibility, and imposes fewer constraints on hardware purchasing decisions and system administration, the userspace offload solution gets consigned to a fairly small niche.
Re:Why do we use the Linux kernel's TCP stack? (2016) by brendangregg on Sunday October 04, 2020 @02:01AM (#24676828)
Right; there's a better presentation I can't find right now, but here's one from Intel showing how XDP was catching up in 2018 http://vger.kernel.org/lpc_net2018_talks/lpc18_pres_af_xdp_p...
Re:Why do we use the Linux kernel's TCP stack? (2016) by LunaSea on Saturday October 03, 2020 @06:20PM (#24675108)
Do you know how far along protocol implementation for higher layers like HTTP and TLS are?
Re:Why do we use the Linux kernel's TCP stack? (2016) by kstenerud on Sunday October 04, 2020 @02:10AM (#24676857)
> These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve.

But is that really such a bad thing? Right now your choices are take it or leave it, or change operating systems. What if we could instead choose libraries suited to purpose, written by experts, like we do with encryption or compression or allocation or any number of common tasks?

You could then have a default implementation suitable for 80% of use cases, and specialized implementations to handle more complex requirements. You could even have per-NIC exclusive device access for a single application!

Re:Why do we use the Linux kernel's TCP stack? (2016) by bogomipz on Saturday October 03, 2020 @06:10PM (#24675055)
I didn't recognize GSQ and Google wasn't much help unfortunately. Could you say a few words about that and also maybe what "pacing" is? Thanks.
Re:Why do we use the Linux kernel's TCP stack? (2016) by brendangregg on Saturday October 03, 2020 @06:41PM (#24675209)
Ah, sorry, that should be GSO (typo!). Generic segmentation offload, which aggregates "super packets" (up to 64 Kbytes) within the network stack to reduce overheads.

Pacing controls when to send packets, spreading out transmissions to avoid bursts that can hurt performance.

Re:Why do we use the Linux kernel's TCP stack? (2016) by ksec on Sunday October 04, 2020 @04:21AM (#24677281)
>The Linux network stack has been rapidly evolving

Are FreeBSD no longer getting any of those? It seems even Network Stack talks from Netflix are now Linux Focused.

It also makes me wonder..... does any of these matter once BPF takes over, quite literally.

Re:Why do we use the Linux kernel's TCP stack? (2016) by brendangregg on Sunday October 04, 2020 @10:02PM (#24683898)
Netflix is lots of Linux and lots of FreeBSD. The Netflix FreeBSD network stack also has various optimizations (we call it the "rack" stack). I've seen a good presentation on how it works, by Randall Stewart, but I don't know that it's public yet.
Re:Why do we use the Linux kernel's TCP stack? (2016) by cpach on Sunday October 04, 2020 @04:58AM (#24677386)
I really like the FreeBSD philosophy but AFAICT the Linux ecosystem moves much much faster than FreeBSD. That means FreeBSD is more stable and has less weird surprises – but also that Linux get new features at a much faster rate.
Re:Why do we use the Linux kernel's TCP stack? (2016) by amelius on Saturday October 03, 2020 @04:13PM (#24674379)
> These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve.

But if you invent them in a different order, you might just end up with a better system ...

Re:Why do we use the Linux kernel's TCP stack? (2016) by takeda on Saturday October 03, 2020 @08:19PM (#24675707)
I don't know... generally when things are developed by someone that doesn't understand the topic end up being quite bad.

Just look at some databases that were created by people who know nothing about databases when they started.