FWIW, Linux itself has plenty of support for TCP offload engines. I don't think Firecracker uses that at the moment, but there's no reason why it has to be that way if that's a true bottleneck in the system.
Re:FreeBSD on Firecracker
by caskstrength on Friday August 25, 2023 @05:42AM
(#37259804)
> Linux itself has plenty of support for TCP offload engines
Could you link to any specific Linux kernel source that implements support for TCP offload? AFAIK networking subsystem maintainers were always opposed to accommodate TCP offload because it is A Bad Idea.
Re:FreeBSD on Firecracker
by packetlost on Friday August 25, 2023 @10:23AM
(#37262070)
I know there's a few userspace networking APIs such as DPDK [0]. Or maybe there's something in KVM code that implements it. If so, it's news to me!
[0]: https://www.linuxjournal.com/content/userspace-networking-dp...
Re:FreeBSD on Firecracker
by yencabulator on Friday August 25, 2023 @03:20PM
(#37265988)
At its simplest, TCP offload can be just letting the hardware chunk a large packet into smaller ones on the wire. I don't think anything trying to offload more than that has really seen much daylight outside of very proprietary "smart NICs".
https://www.kernel.org/doc/Documentation/networking/segmenta...
I think the Firecracker team has stated that PCIe bypass wasn't something they wanted to do, so I don't see how they'd open up to other accelerators' bypass method. But seems like building a vmm from the rust-vmm toolbox is 'not that hard' and there are some PCIe bypass crates already, so... Have fun?
I'm not saying an actual NIC with TCP offload, but instead something like adding TCP offload to the virtio nic if for some reason initialization of the network stack was a latency problem. If the VM is only running at layer 3, most init latency problems disappear.
Re:FreeBSD on Firecracker
by packetlost on Friday August 25, 2023 @01:28PM
(#37264510)
This is exactly what I'm referring to. You have a pool of virtual NICs on the host in user-space created by the VM runtime that get assigned to the guest on provision-time, which just passes through any commands/syscalls (bind/connect/etc.) via a funky driver interface. You'd have to mangle the guest kernel syscalls or libc though, it might be really ugly.
> You'd have to mangle the guest kernel syscalls or libc though, it might be really ugly.
You wouldn't have to. There's patches for hardware TCP offload using the normal socket syscall ABI. The kernel net stack maintainer is pretty ideologically against them so they're not mainlined, but plenty of people have been running them in production for decades and they're quite mature.
Re:FreeBSD on Firecracker
by packetlost on Friday August 25, 2023 @01:43PM
(#37264690)
Oh neat!
Regardless, I'm still surprised microkernels aren't more popular in this space, but perhaps the losing the ecosystem of Linux libs/applications is a non-starter.
Even if the idea wasn't fruitful, the conversation was fun. Thanks for engaging and challenging my bad ideas!
Edit: I've also realized I was thinking of Unikernels, not microkernels and I've been calling it the wrong thing all night. *sigh*