cptnntsoobv

Short answer: yes.

I guess you are thinking of iouring being disabled in dockeresque rumtimes. This restriction does not apply to a VM, specially if you can run your own kernel (e.g. EC2).

Experiment away!

This abuse of notifications (and the user) is quite prevalent, unfortunately.

In the case of the Uber app you should be able to turn these off (they are conveniently enabled by default, of course!).

On the iphone it's Setting -> Communication -> Marketing Preferences -> Push notifications.

For UDP I'd look into GSO/GRO to get an upper bound on what pure kernel can do.

With performance benchmarking, specially in networking there is no end to "oh, but did you think of that?!" :)

> You actually can do a full checksum

Indeed! This is what I had in mind when I wrote "cumbersome" :).

It's been a while for me to be able to recall whether the problem was the verifier or me, and things may have improved since, but I recall having the verifier choke on a static size limit too. Have you been able to use this trick successfully?

> Always build the eBPF programs in a container

That should work generally but watch out for any weirdness due to the fact that in a container you are already inside a couple of layers of networking (bridge, netns etc.).

> IPtables couldn’t do more than 5Gbps of throughput (TCP!)

Is this for a single connection? IIRC, AWS has a 5gbps limit per connection, does it not? I am guessing since you were able to get to ~10 it must be a multi connection number.

XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution:

* The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug

* For the same source code, the code generated by the verifier can change across compiler versions in a breaking way, e.g. because the new compiler version implemented an optimization that broke the verifier (see https://github.com/iovisor/bcc/issues/4612)

* Checksum updating requires extra care. I believe you can only do incremental updates, not just because of better perf as the post suggests but also because the verifier does not allow BPF programs to operate on unbounded buffers (so checksumming a whole packet of unknown size is tricky / cumbersome). This mostly works but you have to be careful with packets that were generated with csum offload, don't have a valid checksum and whose csum can't be incrementally updated.

As the blog post points out, the kernel networking stack does a lot of work that we don't generally think about. Once you start taking things into your own hands you don't have the luxury of ignorance anymore (think not just ARP but also MTU, routing, RP filtering etc.), something any user of userspace networking frameworks like DPDK will tell you.

My general recommendation is to stick with the kernel unless you have a very good justification for chasing better performance and if you do use eBPF save yourself some trouble and try to limit yourself to readonly operations, if your use case allows.

Also, if you are trying to debug packet drops, newer kernels have started logging this information that you can track using bpftrace which gives you better diagnostics.

Example script (might have to adjust based on kernel version):

    bpftrace -e '
        kprobe:kfree_skb_reason {
        $skb = (struct sk_buff *)arg0;
        $ipheader = ((struct iphdr *) ($skb->head + $skb->network_header));
        printf("reason :%d %s -> %s\n", arg1, ntop($ipheader->saddr), ntop($ipheader->daddr));
    }'

> Saves the full VM state to disk

Does this include the RAM for the VM? For auto-idle systems like this where to park the RAM tends to be a significant concern. If you don't "retire" the RAM too the idling savings are limited to CPU cycles but if you do, the overheads of moving RAM around can easily wreck any latency budget you may have.

Curious how you are dealing with it.

The spot termination warnings arrive 120 seconds before termination https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-ins...

As the saying at Netflix used to go back in the day: we need to become HBO before HBO becomes Netflix

> One cool thing we learned was how quickly you can Hibernate and wake up x86 EC2 instances. That ended up being a game-changer for us.

Could you talk more about it? Are you keeping a cache of hibernated EC2 instances and re-launching them per request? What sort of relaunch latency profile do you see as function of instance memory size?

It works on AWS's metal instances, doesn't it?

That is definitely a constraint though, as the smallest available metal instances are pretty large, hence costly on a per instance basis.

Fair point about the length leak.

Good luck with this.

I see, pretty cool!

I noticed that when typing in the email box nothing happens in the recording. I assume that this is to hide the email address but it would be nice to have some sort of visual cue (may be asterisks, like passwords?), otherwise it looks broken.

I can see this being useful in places where compliance is important (medical, finance etc.).

Is the blurring not reversible though? Even if lossy?