That's definitely a security professional's read :)

"Isn't supposed to be able to" is a lot longer and distracting vs the oversimplification-for-sake-of-understanding of "can't". As far as it being proven wrong though - that's already happened, eg CVE-2021-29154

https://blog.kernelcare.com/vulnerability/specially-crafted-...

That's fair. I understand that in the ideal world, it would be "can't." I guess my concern is that the wording kind of hand waves away any potential security issues, when people interested in this tech should absolutely be made aware of them.

“is not authorized to” ?

Yes, especially with Spectre. I looked into using eBPF with seccomp (which currently only supports BPF) and the consensus seemed to be that they're not going to add any more eBPF to the kernel and have basically given up on making it secure.

I wouldn't be surprised if it becomes entirely root-only by default soon, if it isn't already.

For others who are curious, https://lwn.net/Articles/857228/

I think they already don't. eBPF has become a favorite jumping point for exploit chains that can very quickly escalate some read/write gadget into full execution.

I would love to read an in depth breakdown of an exploit using eBPF as part of its execution chain. Do you happen to have an example/link?

This was the one I was thinking of: https://googleprojectzero.blogspot.com/2020/

Scroll down to "The ultimate ROP"

The BPF capability should really only be given to root. I don't think it really gives any new attack surface. All I could see is it giving black-hats an easier interface to "kernel-level-fuckery".

I'd recommend to anyone running Linux that they do exactly that - disable unprivileged eBPF. For a server you shouldn't need that, you can just drop privs for your service after setting up the filter.

They should probably put it behind its own capability like CAP_LOAD_EBPF.

Forcing signed ebpf will also help though.

It’s not an easier interface. It’s much easier to write kernel modules than to mess around with eBPF.

That seems wrong. The build environment for eBPF programs is simpler (you don't even need a working kernel tree), and, much more importantly, eBPF programs are constrained and can't crash the kernel, which is easy to do accidentally when writing an freestyle C LKM. You can learn enough to get stuff done with eBPF inside of a couple days; the same is absolutely not true of kernel modules.

I find the verifier to be extremely frustrating. It's also very difficult to debug eBPF programs, especially in regards to why it's not passing verifier. Combine with ridiculous restrictions from BCC like not being able to use functions in macros, or the vagueries of the bpf target in clang and it's extremely frustrating.

I've personally never found obtaining a working kernel tree to be difficult, certainly easier than a working BCC toolchain. Or all of the various compiler flags needed for clang not to emit code incompatible with the verifier.

I don't use BCC or really understand why anyone would; I use clang, and just produce .o's.

The verifier is definitely annoying, especially at first, but I found myself sort of quickly working out the verifier's expected idiom, and a lot of it can be wrapped with macros.

All of this drama pales in comparison to writing freestyle C code in the Linux kernel without causing random panics.

Easier to get started maybe, but there's something to be said for the 'ease' of having to worry a lot less about crashing the kernel you're working on.

I've crashed multiple kernels with eBPF programs. I don't buy this argument at all.

I call bullshit! Ebpf programs cannot crash the kernel since they cannot be run if they contain bugs in the first place.

The kernel, however, can and does have bugs. I've seen several thousand hosts get taken down with a perfectly correct eBPF program due to a buggy kernel.

That assumes that the virtual machine has no bugs

Not so much (eBPF VM bugs are pretty rare, as you'd expect, since the VM is very simple) --- you're much more likely to run into bugs in the C-code helpers the kernel exports. If you're malicious, you can also hit verifier bugs that'll give your eBPF code raw pointers, but I don't think you're likely to stumble on them accidentally.

I said 'less' rather than 'at all' for a reason. I'm not really sure what to tell you if you disagree that it's a lot easier to accidentally crash your kernel with a from-scratch module than ebpf though. That's an experience pretty drastically at odds with mine.

Can you describe some of those bugs? What helpers were you using? I've been doing bonkers stuff with eBPF for the last year and, while I've definitely had bugs, none of them took my kernel down.

I no longer have access to the code, and the kernel has since been patched. Mostly doing some observability around new network connections.

We definitely hit this one at some point: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1763454

We also ran into a couple of similar but unrelated panic bugs on much newer kernels on non-Ubuntu distros.

Those would be considered kernel bugs though, and should be getting fixed when reported. While a crash in a kernel module is just a bug in the kernel module itself

Could you share the ebpf program and the external dependencies that can reproduce the crash?

It'll be valuable to learn this, so that we might be able to proactively address them. Ebpf is the core of our product.

Are you saying this is factually wrong? I assume eBPF is designed to only access certain points explicitly coded in the kernel but the exploits can access any part of kernel. Is this what you are hinting at? Could you please clarify the statement.

There are at least two issues here.

First: eBPF code is JIT'd in the kernel; at runtime, it is simply native code running at CPL0 alongside the rest of the kernel. Running eBPF code working with pointers is... working with raw pointers. There's no interpretation layer to bounds check or otherwise provide safety.

But eBPF code is meant to be safe: you can get a handle to some kernel structure that's passed to you from trusted code, but you can't bounce from it to a random offset in kernel memory. The way eBPF does this is by verifying the CFG of your eBPF program before it's translated to amd64 or arm. eBPF programs are generally just C programs (the simplest and best way to write an eBPF program is just to write a C program and compile it with the right LLVM flags), and verifying C programs is a hard problem; eBPF gets around this by only accepting a subset of all possible programs (those where memory accesses are simple enough to prove safe, that don't jump anywhere outside of known narrow range of program text, and that don't have unbounded loops).

The tricky thing here is that the eBPF verifier is pretty complicated and lives only in the kernel. People have found bugs in it. If you find a good verifier bug, you can launder an untrusted pointer into your eBPF program (in the end, these bugs end up looking sort of like the browser Javascript RCEs that finagle a bad pointer out of some part of the browser API).

The biggest mitigating factor for these bugs is that Linux systems generally don't expose eBPF to any user other than root, so the upside to these kinds of bugs is limited (it gives you root->kernel, which is not nothing, but not the top of most people's priority list).

The other big issue is that eBPF is a huge source of in-kernel flexibility about runnable code. Modern exploit mitigations are in large part about making sure that instructions running at CPL0 are all known, so that if you manage to corrupt allocator metadata or write an arbitrary 8 byte value at an arbitrary 8 byte offset you can't easily turn that into remote code execution. But, of course, eBPF is an in-kernel JIT; it's there to run essentially random code inside the kernel. eBPF code is normally constrained, but if you have a kernel memory corruption bug, you can aim it at the eBPF subsystem and violate the kernel's assumptions.

Yes, "can't" should be replace by "shouldn't".

If there's a physical possibility, it's just a matter of time before someone finds a way, as was proved by the CPU cache bugs leaking information.

"shouldn't" is ambiguous as it might suggest the responsibility is with the the user of eBPF rather than internal to it.

Perhaps "is designed not to" but that's a mouthful.

I think we should accept "can't" and yet know the limitations of certainty.