shivanshvij

We started an Ollama container on a MacBook. There's no NVIDIA GPU, no CUDA toolkit, and macOS doesn't even have CUDA drivers. Ollama found an NVIDIA GPU anyway: A 128 GB Blackwell GPU on a DGX Spark across the network.

Our anpproach enables this by intercepting CUDA calls and forwarding them to a remote server. It takes one command, requires no code changes, and the application has no idea.

Founder of loophole labs here!

Cilium is definitely the gold standard if you’re working with Kubernetes clusters and need a full CNI, but if you want to extend CNI functionality without replacing it, then this approach is the only option.

It works quite well because Cillium (and all CNIs that I’m aware of) don’t use XDP like the blog post mentions, they use Netkit instead which is an alternative to veth designed for netfilter-like use cases.

This means XDP can work alongside Cillium (with enough tweaking) which is what we wanted to be able to do.

If you’re using pure containers and no CNI, then of course this provides a significant speed up even beyond netkit devices.

Sorry about that! It's fixed!

Sorry about that, we've fixed it!

That's a great point. This was one of my iPerf3 runs at one point:

`iperf3 -c 172.31.45.187 -p 5201 -P 128 -t 5 -b 512M -u -l 1448 --bidir | grep "\[SUM\]\[\(TX\|RX\)-C\].*receiver"`

We're also looking at using packet generators to test raw packet throughput. There's a lot more bottlenecks we can cleanup I'm sure.

It is! We demo'd it on stage at KubeCon last hear: https://loophole.sh/kc2024

Wow thank for the recording! We were able to push out a fix, should be workin g now!

No this was multiple connections - and we tried with both `iperf2` and `iperf3`, UDP and TCP traffic. UDP actually does much worse on `iptables` than TCP, and I'm not sure why just yet.

We absolutely ran into these issues.

A couple notes that help quite a bit:

1. Always build the eBPF programs in a container - this is great for reproducibility of course, but also makes DevX on MacOS better for those who prefer to use that.

2. You actually can do a full checksum! You need to limit the MTU but you can:

  static __always_inline void tcp_checksum(const struct iphdr *ip_header, struct tcphdr *tcp_header, const __u16 tcp_len, const void *data_end) {
    __u32 sum = 0;
    __u16 *buf = (void *)tcp_header;
    ip_header_pseudo_checksum(ip_header, tcp_len, &sum);
    tcp_header->check = 0;
    __u16 max_packet_size = tcp_len;
    if (max_packet_size > MAX_TCP_PACKET_SIZE) {
        max_packet_size = MAX_TCP_PACKET_SIZE;
    }
    for (int i = 0; i < max_packet_size / 2; i++) {
        if ((void *)(buf + 1) > data_end) {
            break;
        }
        sum += *buf;
        buf++;
    }
    if ((void *)buf + 1 <= data_end && ((__u8 *)buf - (__u8 *)tcp_header) < max_packet_size) {
        sum += *(__u8 *)buf;
    }
    tcp_header->check = csum_fold_helper(sum);
  }

With that being said, it's not lost on me that XDP in general is something you should only reach for once you hit some sort of bottleneck. The original version of our network migration was actually implemented in userspace for this exact reason!

Oh no we absolutely have to also correct the IPv4 header checksum!

Awesome we’ll be looking forward to it!

Actually the latest benchmarks were ran on a Fedora 43 host, which as far as I can tell uses the nftables backend for iptables!

As far as we can tell, it’s a mixture of a lot of things. One of the questions I got asked was how useful this is if you have a smaller performance requirement than 200Gbps (or, maybe a better way to put it, what if your host is small and can only do 10Gbps anyways).

You’ll have to wait for the follow up post with the CNI plugin for the full self-reproducible benchmark, but on a 16 core EC2 instance with a 10Gbps connection IPtables couldn’t do more than 5Gbps of throughput (TCP!), whereas again XDP was able to do 9.84Gbps on average.

Furthermore, running bidirectional iPerf3 tests in the larger hosts shows us that both ingress and egress throughput increase when we swap out iptables on just the egresss path.

This is all to say, our current assumption is when the CPU is thrashed by iPerf3, the RSS queues, the Linux kernel’s ksoftirqd threads, etc. all at once it destroys performance. XDP is moving some of the work outside the kernel, while at the same time the packet is only processed through the kernel stack half as much as without XDP (only on the path before or after the veth).

It really is all CPU usage in the end as far as I can tell. It’s not like our checksumming approach is any better than what the kernel already does.

Hi HN, Shivansh (founder) here, happy to answer any questions folks might have about the implementation and the benchmarks!

Hey Everyone,

Founder of Loophole Labs here! The team and I are happy to answer any questions you might have about Scale Functions or Loophole Labs in general!

We wrote fRPC because we really liked the DevX and tooling around the proto3 syntax, but we needed the generated code to be significantly more performant than what gRPC provides.

We also needed the ability to extend the RPC framework with other messaging patterns (like pub/sub) and we needed to be able to reuse the underlying TCP connections as required.

There's no way to do this with gRPC (or any other RPC framework that I'm aware of), and so fRPC was born!

As for streaming, it's not only planned (you can check out our roadmap here: https://frpc.io/getting-started/roadmap), but we've already got a PR open for it (https://github.com/loopholelabs/frpc-go/pull/2). Just need to work out a few more bugs before we merge.

Hey Everyone,

Founder of Loophole Labs here. The team and I are happy to answer any questions you might have about fRPC, Frisbee, or Loophole in general!