Excellent article all the way around. I may assign this to my networking students further into the quarter. It provides a nice alternative use case that should help them think more critically about the relationship between applications, the socket API, and the rest of the network stack. Thanks!

This is great, thanks for sharing. I'm curious about the downstream proxy process (i.e. ::1234) and how you scale it and balance load across multiple instances of the process. You can't really use iptables to load balance your processes as either the DNAT or REDIRECT mechanism will modify the destination address, right?

Ex. # TPROXY directs all traffic to :1234, and these rules load balance to 4 different processes

iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 0 -j DNAT --to-destination 127.0.0.1:8080

iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 1 -j DNAT --to-destination 127.0.0.1:8081

iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 2 -j DNAT --to-destination 127.0.0.1:8082

iptables -t nat -I OUTPUT -p tcp -o lo --dport 1234 -m state --state NEW -m statistic --mode nth --every 4 --packet 3 -j DNAT --to-destination 127.0.0.1:8083

We have a single Accept queue for all the ports. For TCP it doesn't create any problems - the new connection rate is rarely significant.

For the accept-queue load balancing see these blog posts:

https://blog.cloudflare.com/the-sad-state-of-linux-socket-ba...

https://blog.cloudflare.com/syn-packet-handling-in-the-wild/

Wow, these are some great resources. Thanks for sharing! I have a call with one of your colleagues in 5 minutes ;)

I have a vpn for myself and some friends that accepts connections on all ports. I set it up over a decade ago on OpenBSD with a simple pf redirect. I have never had any problems with it, but it obviously doesn't see nearly as much traffic as Cloudflare.

Does OpenBSD handle this differently than Linux, or am I doing this wrong?

The trick is that they want the application to be able to see what the original destination IP and port were. I'm not sure if a pf redirect preserves that information.

It does -- or appears to, at least, for my instances of (OpenBSD's) spamd.