Re:Make your own VPN with Fly.io, tailscale and GitHub
by Tor3 on Thursday May 25, 2023 @05:06AM
(#36068505)
The problem with TCP compared to UDP is when you do VPN over links where there's some amount of round trip time. I routinely run VPN over a 300+ ms ping times, and any TCP-based VPN suffers dramatically when doing a TCP connection through that TCP-based VPN. Switch to a UDP-based VPN and the problem disappears (easy to test by using OpenVPN and switch the configuration between TCP and UDP. But I've tested with other types of VPN as well). When you're closer to home the problem disappears.
Re:Make your own VPN with Fly.io, tailscale and GitHub
by klabb3 on Thursday May 25, 2023 @04:18PM
(#36075624)
Thanks, yes this makes a ton of sense.
Another thing I wondered about is how much CPU overhead VPNs add, and how it performs when maximizing throughput. (Not Tailscale but a “regular” one with kernel packet switching). Do you have any experience with that?
Re:Make your own VPN with Fly.io, tailscale and GitHub
by Tor3 on Friday May 26, 2023 @02:53AM
(#36080900)
I only use OpenVPN regularly. My internet connection is either 100Mbit or 1Gb both up and down, but depending on where I am the actual external bandwidth varies - mostly it's around 50Mbit end-to-end, subjectively. If I send data at max speed through the network then I observe that OpenVPN may use quite a bit of CPU (maybe up to around 40% of one core (i7-7500U), but it doesn't limit the transfer speed I get compared to when I do a direct transfer without VPN (interestingly, on long latency lines (when I go OpenVPN from Japan to Europe) I often get better and more consistent performance when going through OpenVPN (configured to use UDP).
Re:Make your own VPN with Fly.io, tailscale and GitHub
by benmmurphy on Thursday May 25, 2023 @08:15AM
(#36069790)
usually VPNs in linux push IP (TUN) or ethernet (TAP) frames between the nodes so you really need to be using UDP or else you are going to have problems with running TCP over TCP and the congestion algorithms conflicting with each other. openvpn which supports TCP refers to this problem as TCP meltdown and advise using UDP where possible (https://openvpn.net/faq/what-is-tcp-meltdown/) VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead of the IP packets but you would still be left with issues when you are transferring UDP across TCP which is not ideal.
Re:Make your own VPN with Fly.io, tailscale and GitHub
by klabb3 on Thursday May 25, 2023 @04:16PM
(#36075595)
Thanks! I don’t have a lot of experience with VPNs but for sure UDP is much better suited for packets which is the abstraction layer that VPNs operate on.
> VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead
Yes! An interesting observation is that TCP composes really well, ie relaying works excellent. However, for VPNs it’d be nesting TCP which melts down quickly.
Compared to another VPN? I’d be curious to know whether the kernel mode byte shuffling solves that problem. But even so, a kernel module is a pretty big ask only for connectivity.
In my experience, UDP in general isn’t as performant in practice as one would think. Not saying you can’t push the limits, and even outperform TCP, but to do so with a reliable cross platform way isn’t exactly trivial today.
All my benchmarks (albeit user space) have shown that pushing bytes over udp has a higher CPU overhead, and that’s even if you omit retransmission, congestion control, etc etc (ie just push garbage bytes). And even if your cpu can handle the throughput, the congestion control can still bite you for god knows what reason. When I ran quic benchmarks they got deprioritized in the presence of tcp traffic. Don’t know all the reasons why (sorry, just didn’t have the time) but at least to me TCP wins the bang-for-the-buck-throughput-on-commodity-hardware category, hands down. Maybe this changes with platform-specific optimized vectored IO, but that alone would be a huge effort. No, the more time I spent on it, the more I appreciated all the things TCP gives me for free, and it’s remarkable resilience in complex conditions. I am also happy I don’t have to worry about bulky 3p libraries. This is what the OS is supposed to do, imo. So this UDP-hype-renaissance we’ve seen over the last years is a bit premature or at least not as obvious as people hoped (including myself).
Another fun fact (for anyone who read this far): contrary to popular belief p2p TCP isn’t harder to do than UDP, not really.