lmb

    Location: London, UK /  Munich, Germany
    Remote: No, hybrid only
    Willing to relocate: Maybe
    Technologies: Go, C, eBPF, Linux
    Résumé/CV: on request
    Email: [email protected]

Systems engineer with ten years of experience building performance critical software. The past couple of years I’ve been maintaining the most popular Go library for working with eBPF (used by DataDog, Microsoft, etc.) and have contributed to the Linux network stack, the Go standard library and other high profile open source projects. Before that I built Quicksilver (global distributed db) and Unimog (anycast L4 load balancer) at Cloudflare. In my spare time I’ve dabbled in embedded operating systems, virtual machine monitors and emulators.

I’m self taught, learn by doing and enjoy going deep on a specific subject. After working as a fully remote open source maintainer I’m looking for a small team that values collaborating in person, in a strategic role owning a domain or problem space, with a view to leading a team.

I’m curious about problems that require exceptional performance and / or reliability in areas like power infrastructure, renewable energy, operating AI models at scale, non-LLM AI, hardware.

Counter point from Gleixner which is worth reading for us in the peanut gallery: https://lwn.net/ml/linux-kernel/87r11qp63n.ffs@tglx/

Missing Allen (2001) by my late father Christian Bauer

https://www.youtube.com/watch?v=kOXSI_vcqK8

Around the turn of the century, my dad's good friend and longtime collaborator Allen Ross vanishes from one day to the next, just after they have finished shooting a film about the Mississippi. Years later, my dad returns to the US to find out what happened to his friend.

It's his most personal film for sure, and I remember him going off to the US for weeks and faxing us letters to keep in touch. It's also the one that had him most scared, he took out life insurance before he left because of the people he was looking into.

We ended up with fasthash64 and lookup3 by looking for a fast hash that is easy to port to the restricted subset of C supported by eBPF with minimal changes. https://github.com/rurban/smhasher is a great resource for that.

I would probably choose different, more robust hash functions if I was targeting regular C.

I think we tried siphash at some point, unfortunately it's significantly slower than the two hash functions we ended up using.

Count min sketches are really neat. A colleague at my old company used them to implement DDoS mitigations in eBPF and wrote it up: https://blog.cloudflare.com/building-rakelimit/

The code is also open source, and I've improved on it a bit: https://github.com/lmb/socklimit Not production ready but a cool idea and implementation.

We don't use kernel bypass anymore, it's all XDP: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig...

What does tail latency for the Zig pool look like? It seems like any Task that ends up in one of the overflow queues will stay there until at some ring buffer is emptied.

Put another way, during a period of more push than pop some tasks may see a very long delay before being worked on?

What does tooling / debugging for p4 look like? How do you inspect what happens inside a p4 dataplane?

Context: I work on a L4 load balancer written in ebpf / XDP. Debugging that is still too hard even thought it's all software and open source.

There is a fair bit of co-evolution of the Linux BPF verifier and the BPF llvm backend. This means that the verifier is biased to do well on bytecode that is commonly emitted by clang. I can imagine that other programming languages will run into the verifier rejecting their output if it differs to much from common clang output. (There is redbpf which compiles rust to BPF AFAIK, but it still uses llvm. There is also a gcc BPF backend, but I don't have experience with that.) The good news is that Linux upstream is receptive to bug reports!

Seems like the right move from a volunteer run project, what will the future will hold though? Artificial scarcity is always a problem.

On another note, for just 20k$ I can offer you exclusive use of the xxgfzrf.dinglebop.me Public Suffix so that you can keep tracking your users. Please reach out to [email protected] if you are interested.

You're arguing that the majority of people use the term differently than you and are therefore wrong. Sounds like the definition of technobabble to me.

Total shot in the dark: I had a similar problem, it turned out the power supply for the router was faulty.

Can you define ready? The functionality of the library is solid, it's based on code we're using in production. We've not committed to a stable API however.

(I'm one of the maintainers of said library.)

In some of the code snippets, the following is used:

  threshold = -bound % bound

I don't understand how that isn't always zero. What am I missing?

The problem is that different kernel configurations end up changing the layout of important data structures. Basically, you need to change the offsets that the eBPF uses.

There is another proposal that uses BTF to do the fixup at load (not compile) time, but that ends up having it's own complications.

It doesn't give any specifics. "2 of 121 parameters were configured incorrectly". Maybe host and port? ;)

What do you mean with this? Refuse ANY is now a proposed RFC https://datatracker.ietf.org/doc/rfc8482/ How is that a band aid?

We've been using LMDB at Cloudflare to store small-ish configuration data, it has been rock solid.

Thank you and the rest of the contributors for such a great library.

LMDB is not a great fit for something like the WAL, where new data is written at the end, and old data discarded at the start. It leads to fragmentation (especially if WAL entries are larger than a single page).

Can you give an idea how big your cluster is? 10, 10s, 100s?

I'm curious how well Normand works in reality, but it's hard to find ppl running it.

The byte code is used as a target for verification, and is designed to be easily JITable on x86 and arm.

There are network cards which offer eBPF offload, but they only implement a subset of features available in the kernel. I imagine that keeping up is hard, since the technology is moving quickly.

Netronome seems to have the most advanced eBPF offload available.

A colleague of mine did a performance deep dive as well: https://blog.cloudflare.com/arm-takes-wing/

It's sad to see Qualcomm ditch such a good platform.

Ok, I've looked at the code, and based on that I think what I said in my original post is correct.

GLB is more complex, but will maintain connections in more circumstances.

If the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.

I haven't looked at what Google has released, but there are big differences between GLB and Katran. (Not affiliated with any of those companies)

In terms of technology, Katran uses XDP and IPIP tunnels, both upstream in the Linux kernel. GLB uses DPDK, which allows processing raw packets in user space, and Geberic UDP encapsulation + a custom iptables module. Neither DPDK nor the module are upstream.

There are architectural differences as well. Katran is much closer to a classic load balancer, and uses connection tracking at the load balancer to know where to send packets for established flows. GLB has no per-flow state at the load balancer, which gives it the very nice property that load balancers can be added and removed from an ECMP group without disturbing existing connections. There is an academic paper about a system called Beamer, which most likely influenced GLB (or maybe the other way round?). It's a good read, and relatively short.

Finally, Katran is really a C++ library you could build a load balancer on, while GLB comes with batteries included.

I think GLB looks nice, hats off to GitHub for open sourcing it.

That repo encourages mindless microoptimization. Don't waste your time worrying how to iterate a slice. Most of the time it makes no difference.

It is possible to pass file descriptors using os/exec, using the ExtraFiles attribute [1]. No need drop down to os.StartProcess.

1: https://golang.org/pkg/os/exec/#Cmd

There is so called "bulletproof" hosting which usually includes being able to spoof IP. So yes, there are economic incentives.