apenwarr

(Tailscale cofounder) Yes, that’s most of it. Most packet-level optimizations are necessarily platform specific and this post was mostly about the linux dataplane.

That said, the architectural multiqueue improvements and memory optimizations are useful starting points to build on for every platform.

(Tailscale cofounder) Nothing to announce yet, but yes, we absolutely do sponsor wireguard, with money and with code.

(Tailscale cofounder) Many years ago I made a VPN for experimental purposes that kept IPsec’s data plane but abandoned IKE. It’s actually a pretty good arrangement and might be applicable here.

My implementation relied on the extremely finicky Linux kernel IPsec and would never have worked in userspace macOS or windows. But yes, a Tailscale control plane with a per-node switchable data plane, one of which is IPsec with PQC, is a real option that would work.

By locking down the control plane it becomes incompatible with classic IPsec, but also avoids most of the security gotchas that plagued IPsec.

(Tailscale cofounder) I’m a little more optimistic; wireguard was always going to need a v2 eventually, that’s just the nature of cryptography. It’ll always be simpler than IPsec as long as it avoids live negotiation (you need to specify your suite up front for each node you talk to; v1 and v2 are the only suites) and continues to go over UDP instead of IPsec’s “it’s a different transport protocol!” madness.

Things like Google’s PSP already achieve that while still being hardware acceleratable: https://github.com/google/psp

Post-quantum negotiation will kinda suck but you don’t have to sacrifice everything.

(Tailscale cofounder) That’s a good callout on DoH support, thanks.

That said, note that if you run your own DNS server on your tailnet, the regular UDP DNS is automatically private because it’s carried over Tailscale. That’s the most common setup for non-SaaS DNS servers. DoH doesn’t really add anything in that arrangement. (And it’s more fiddly because you need to get and refresh a TLS cert.)

(Tailscale cofounder) I see a few comments here that using kernel wireguard would make it faster; it’s not really that simple. In fact, for a while (and we wrote a blog post about it), our optimizations made wireguard-go faster than kernel wireguard because it was better optimized. They adopted some of those improvements and now we’re on to the next order of magnitude together.

For really high bandwidth cases, things like DPDK are the long term best choice and are primarily userspace, for good reasons. Kernel mode is not the pure benefit it once was (if it ever was).

Separately, wireguard itself has a problem that the crypto suite it uses is not supported by hardware accelerators. So if we want to get into the hundreds of gigabits range, we will possibly need to switch packet formats entirely. (But, wireguard also needs to update to support post-quantum so maybe they’ll fix both problems at the same time and we can join in.)

Is your theory that "any competent security review" will find every security hole in a product? Because that sure would be great if it were true. Unfortunately it does not match my experience.

(Tailscale CEO) I don't know what to tell you. The problems that are found internally, or via security reviews and pentests we pay for, are ones that we fix before releasing. They don't need bulletins.

Bugs that are found by other people are found, by definition, after release. They are therefore more likely to need a bulletin.

(Tailscale CEO) You have posted here multiple times that "none of the code has had a security audit" and that the SOC2 audit "is not the same thing."

It's true that those two audits aren't the same thing. However, the SOC2 auditor confirms, in the published report, that Tailscale has regular and ongoing security audits including penetration tests and many kinds of code reviews.

The security audit report, which you perhaps imagine to be a long list of vulnerabilities... doesn't look like that. It says we don't have a long list of vulnerabilities. The security bulletins are all here: https://tailscale.com/security-bulletins

I’m glad you liked it!

Japanese manufacturing addressed this too! There’s a whole branch of Deming’s work around supply chains and how it’s often worth working with “more expensive” suppliers when their quality is consistently high, because compensating for low quality inputs to your own work is far more expensive.

(Tailscale founder here) Two main differences: first, every DERP server used by your tailnet must be accessible by every node on your tailnet at all times, otherwise you get hard-to-debug netsplits. That's a very high bar to maintain so we've historically recommended you don't try. In contrast, peer relays are "if a given pair of nodes can connect through any of the relays, go for it" so deploying one is always a performance and reliability improvement.

Secondly, peer relays support UDP while DERP is TCP-only. That would be fixable by simply improving the DERP protocol, but as we explored that option, we decided to implement the Peer Relay layer instead as a more complete solution.

[Tailscale CEO here] I see a lot of comments asking why Tailscale would branch away from our "core product" and build this thing that seems unrelated at first. One answer is that just about every single Tailscale customer (or homelab user!) is dipping their toes into AI right now, and they often come to us and ask how to integrate their stuff into Tailscale. Aperture is our answer to that.

A separate goal I have personally: demonstrate that anyone can build really neat stuff directly on top of the "Tailscale platform." One of my rules for the Aperture team was, you're not allowed to change core Tailscale, you have to build entirely on top as if you were some partner company. So this is a demo of how anybody can make pretty slick, easy-to-use, and yet highly secure stuff by building on Tailscale (the open source packages, or the commercial product, or both).

(I'm a Tailscale employee) The recent versions of the Tailscale k8s operator actually used a pre-release of the Services feature to do exactly that. So, not much difference. The official Services release is making that functionality available for more use cases (and generally better documented and user friendly).

You’re right except in the very specific case of the App Store purchase or download process. You only get one chance at FaceID and then it demands a password. But, if you cancel and do it again, you get another chance at FaceID. It’s mystifying why they’d make that UX choice.

(Tailscale cofounder here) Tailscale already gives every node on every tailnet a globally unique internal IPv6 address, that is reachable even if you don't have IPv6 on the "outside" network. If your apps and OSes are all willing to use IPv6, you haven't had a problem since the early days of Tailscale; they've been solved for years.

Alas, the "apps and OSes are all willing to use IPv6" problem is a persistent one, so we have to make IPv4 work too.

It’s possible to build a cache without any insight, but to make a cache that works well in a given domain requires knowledge from outside the world of caching itself. The same is true of summarization and compression.

Tailscale now supports custom OIDC providers. But if you already have the ability to host one, you won’t benefit from what’s in the above article (which is about hosting stuff at home even without a public IP address). https://tailscale.com/blog/custom-oidc/

I recommend against using Funnel for this use case (because it exposes your server to everyone in the world, not just your trusted users). Tailscale node sharing is free and secure for private networks of friends, and there are lots of people using it with Minecraft: https://tailscale.com/kb/1084/sharing/

To answer the question in another thread, node sharing also works with UDP. (Funnel is TCP-only due to the vagaries of IP addresses and TLS certificates when facing the outside world, sigh.)

(Interviewee here) WireGuard itself barely touches DNS, and tailscale as far as I know doesn’t have any code that would change how your external DNS resolution works. What you’re seeing might be a misdiagnosis. If you email tailscale support we’ll be happy to help figure out what happened and if it’s a bug.

Exit nodes are usually still behind your firewall and have no open incoming ports. If you're willing to reconfigure your firewall to open incoming ports, you probably didn’t need Funnel in the first place.

I haven't played with OpenWRT SQM for a while, but if it's easy to reproduce with gfblip, it might be a simple matter of telling them your exact SQM settings and the URL to try.

Chances are changing the SQM backend (eg. between fq-codel and cake) will at least change the behaviour and likely make the bug go away.

In general, the load generated by a series of these pings is so low as not to matter, unless a whole ton of people start doing it at once. But in that case, gfblip's trivial backend code will ask the frontends to slow down so that aggregate load stays low.

I've been on the Internet for a long time, including doing Debian stuff (popularity contest!) back in the 1990s. I don't know what attrition.org is though.

Oh neat! If you see occasional red blips, you have probably found a bug (or misconfiguration) in your OpenWRT SQM. gfblip doesn't lie. In fact, I created it to find annoying router bugs like this. If you click the "DNS" checkbox you can find all sorts of terrible router bugs.

It's not too surprising that you don't see latency increase under light load like Youtube. "Under load" really means uploading (especially) or downloading (sometimes) at maximum speed. But Youtube usually doesn't download at maximum speed, because that would imply you can't keep up with its streaming video rate, and you'd get glitches. So your connection is likely not really "under load" at that time. Try uploading a large file somewhere and you should see an immediate change in blips.

However, the preflight check is sufficient to check latency :)

"Preflight check" is such a wrong analolgy, since with CORS you fly all the way to the destintion to check if you're allowed to fly to the destination.

You might mean if you turn "on" HTTPS-only mode?

The silly trick blip uses involves pinging non-encrypted HTTP web servers, which is not allowed from an encrypted web page. So you really have to load blip from a non-encrypted server.

This code is from 10 years ago. Most likely we could find some way to work with HTTPS nowadays, but alas, I don't have time to maintain it.

When you ping to the phone, you're probably running into the phone's power saving feature. When the phone generates the request, it stays powered up for a bit until the answer comes back, which is why you're not seeing the same random delay from blip.

"https with auth" is fine and good, and obviously the world has been heading in this direction. But I secretly suspect this is because 90%+ of developers nowadays don't know how to hack on any layer below http.

Tailscale is not a typical VPN; it's just a system that attempts to provide beyondcorp-like behaviour at a lower level of the stack, so that you don't have to rewrite all your apps (ssh in this case!) to use https, and don't have to have open ports in your firewall, and don't have to run everything through the cloud if you don't want.

As in my post above, there's more than one way to do it. You can also build traditional-beyondcorp-over-https on top of a Tailscale network, so you get all the improved network connectivity and also all the benefits of a "pure" beyondcorp architecture.

(I'm a tailscale cofounder) I think of Tailscale more like a set of tools that lets you do any architecture you want. Nobody has to use Tailscale ssh console, but if you believe in the future of wasm -> apps -> web console -> ssh, now you can have it.

On the other hand, if you believe in the future of OS private network connectivity -> console -> ssh, then you had that already with native Tailscale and Tailscale ssh.

If you believe in OS private network connectivity -> browser -> javascript console -> ssh, then you can do that too, by installing tailscale in the native OS and then the browser can use it.

I actually agree with you, I'm very suspicious about a world where we just move everything into the web browser. But on the other hand, sometimes it's really handy to have that option.