I tinkered with cloudflare before that but just couldn’t get on with the interface of the admin tooling.

With Tailscale I have a lot more confidence that I’ve set up the access rules as I need them. It’s all just a lot more obvious.

That is something I have really appreciated about Tailscale. It seems to consistently not mess with the existing environment. Considering it does networking witchcraft and it works on a variety of architectures and OSs this is quite an accomplishment.

I suspect Tailscale's customers have found the same.

This is one of the major things I _don't_ like about Tailscale. I wish they'd just stick to enabling Wireguard and making the authentication easier (i.e., where they started). I'm not a fan of most of the features they've added since. I don't want service discovery, magic DNS, SSH key management and/or the kitchen sink bolted on.

Linux DNS is a clusterfun: https://tailscale.com/blog/sisyphean-dns-client-linux/

But, yeah, without systemd-resolved Linux DNS is a fight for the death between uncooperating processes. NetworkManager is okay but there are a dozen buggy variants in the wild we have to work around.

Linux is by far the worst platform for DNS config.

I totally recommend systemd-resolved. It's the only thing that does DNS well on Linux.

[1]: https://en.wikipedia.org/wiki/Name_Service_Switch

The firewall is the system. Just like apple bypass its own firewall and just send packet back home. Or the chinese way.

Of course as said by one of the author the key is to control port 22 or rule for ssh. That is not a totally lost. Still, one that is ok … you are breaking the system by promoting a way to bypass it. Or just 1 rule. It is so hard to remember.

While it might seem logical in your mind to bolt on extra features and add value, your customers evaluate risk based on functionality of the software they are approving. Customer buys a VPN solution, magically gets remote access that bypasses firewalls. Can we trust Tailscale to not roll out a remote file backup feature and start silently exfiltrating data (as an extreme example)?

(1) a target server needs to run "tailscale up --ssh" to enable the SSH server

(2) your Tailscale ACLs have to permit it. Our default, if you've never set your ACLs (as is usually the case for personal users), is that you're allowed to SSH to your own untagged devices only.

For an org that's already using ACLs, you won't have any SSH rules defined and thus nobody in your org can enable the SSH server. (Or rather, they can enable it but nobody can connect to it.)

If your concern an org that's using the default "all packets are allowed" ACLs?

I can't speak for mike_d specifically, but there is a concern with having (potentially significant) modifications made to the codebase that aren't surfaced in the release notes. I imagine closed-source projects do this on a regular basis whether customers know (or care) or not.

The expectations for opensource projects are different though, particularly when it comes to system-level or near system-level components. So not being able to access the functionality is a great default but it doesn't address side effects of the changes or the desire to know about changes being made in our environments.

Until you decide to ship a completely on-prem Tailscale server, ACLs mean nothing. They can be modified by the same rogue employee that added an SSH server that bypasses local firewalls to our environment without telling anyone.

(We don't use Tailscale SSH, and are unlikely ever to; we have a separate source of authentication truth for SSH, and a separate certificate-based access control system.)

My hope was that with a little public prodding they would do better in the future. It is a product I want to like, maybe not for what you or I do, but lots of folks out there are slinging cat pictures where it will be a net benefit.

This is a postmortem-worthy incident on Tailscale's part.

Now what I have to do is file a bunch of tickets and take a bunch of meetings to get a block removed from the overall site. Really, what I was trying to do is provide nformation to the Tailscale developers that enterprise already considers their website/product scary enough to do a whole block, and if they want to expand into enterprise, they may want to understand the reasons for that.

Not all large enterprises are this disfunctional. I'm sure Tailscale are doing just fine.

Anyways, I'm just here to say, corporate security teams are definitely not OK with you doing a rogue Tailscale install, and that's as it should be.

You might be shocked at how often I get "can you deploy a tarsnap server on port 443? My company's security team won't let me connect to your server on port 9279" requested.

I mean, it's trivial to bounce the TCP connection... but I'm not going to help subvert security policies.

The fundamental problem with the approach really is that connections are different over the tailnet and over the local network. Here is a specific use case that is painful:

1. There exists a cluster of machines, each with large amounts of locally attached storage. They are all on the same local network and connected with 10Gb (and likely soon 40Gb ethernet interfaces).

2. Each machine is individually on the same tailnet so they can be accessed remotely.

3. Remote users frequently need to move large amounts of data between machines. A user copying a few hundred gigabytes of data with "scp" is normal.

4. For performance reasons, it's preferred to avoid the Tailscale/wireguard overhead when copying data between adjacent machines in a rack.

At this point, if I enable tailscale ssh for remote login, it appears that the problem of key management for connections between local machines (using ssh over the normal interface, not the tailnet) still remains, and in fact, the overall authentication configuration is more complex than it was before.

What I would love to exist, and would make me instantly use this feature, is if the tailnet issued SSH certificates (probably injected into its own ssh-agent?), the existing tailscale SSH implemention worked just like it currently does (it's great!), AND I could manually configure servers to accept certificates issued by the tailnet. Then SSH paths like "laptop --> (over tailnet) --> server 1 --> (over local network) --> server 2" could be made to work transparently, for those machines that need it, and for regular users, it still "just works".

But so does regular SSH over Tailscale, so Tailscale SSH isn't special in that regard.

> So it doesn't matter whether you have other processes (or iptables rules, etc) that would prevent the Tailscale SSH server from binding to port 22.

This sounds like a great feature when exploiting buggy WordPress/php apps! /s

I realize this is a feature - but it's a bit sad that the standard package handling isn't up to the task; leaving (I expect) the tailscale daemon as a "magic" netcitizen - not featuring in neither "ss" or "iptables" output (why can't I login to opensshd?).

[edit] It boils down to principle of least surprise, managing expectations, etc. - proper documentation is indeed key.

(I'm not impartial about Tailscale.)

Humility helps a lot on the internet- the important thing about iptables is that it runs on millions, possibly billions of machines. Production systems that don't have unit tests but run at scale aren't worse than systems which are newly introduced but have fairly unknown implications.

I'm not making a value judgement about people who need to keep using iptables. I might be making a value judgement about people who demand that everyone else keep using iptables.

But, my point still stands. You can't simply assert your system is better, it has to be proven at a scale similar to iptables before you can say that.

(I'm not impartial about Tailscale, but I don't work there).

Well, that's certainly different from "all TCP port 22 packets" - I suppose some emphasis should be on "after the WireGuard decryption" (ie: over the wireguard interface). It's not entirely clear from the comment (but probably clear to engineers working on the tailscale code).

I read it as if tailscale snapped up packets before the kernel from (all) network interfaces...

I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes?

Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs, especally if I could add "check" to the ssh. agent pam module?

One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10.

I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them.

Again, Tailscale SSH looks very nice, job well done!

Minor suggestion, for future and new users, is it possible to get a calculator where you could input the number of users you expect, the number of servers you want to include, expected unique ACL's and provide you an ETA of what your license cost would be?

Linux-only on the server right. macOS support is kinda there (in git) but not entirely done and not included in the GUI builds. Windows server support is tracked in https://github.com/tailscale/tailscale/issues/4697.

You can use any SSH client from any OS.

> Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs

Some of the start of that is in https://github.com/tailscale/pam

> One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10.

You only pay for unique humans, not tagged role account devices. I wonder if your email got eaten as spam or something. Email me (username at tailscale) and copy sales@ and I'll make sure somebody replies. But I don't think you need a custom plan.

> I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them.

Hey, if you've got something that works, stick with it. :)

Thanks for the info about pricing, I set up the 1 user free account and started that to get some hands on experience, and I'll copy you on pricing if I can't get it figured out. Thanks!

I've tried this earlier and was unsusccessful sshing from my iPad, using Termius and Blink apps. Not sure if there are specific client requirements on the iPad?

Can you file a bug with details of what you saw? Either https://github.com/tailscale/tailscale/issues/new or email support@ ... whichever you're more comfortable with.

I had a notification asking me to verify, but because of Focus, that notification didn't show up anywhere that I could see...... So in theory, this should work, will try again.

With SSM, I can easily run an agent on every instance. Tailscale has pretty tight device limits on the Team and Business plans. I have no idea what the custom pricing looks like, but I'm guessing it would exceed my budget. What's the intended way to use this with a large number of servers? A small team can easily have more devices than 5x or 10x the number of users. Should we just set up some "gateway"/"bastion" instances to access via Tailscale SSH and then use regular ssh from there? Some sort of more limited device mode that doesn't count against the device limit (for ssh only, perhaps?) would be great.

Does tailscale have the same issue?

https://github.com/cloudflare/cloudflared/issues/574

Cloudflare have ignored the github issue (which includes a solution) but at least 3 other people seem to have found my solution helpful.

Nevertheless, I had to open SSH on each machine, and it's a nightmare to close up the firewall so only Tailscale gets through. You'd think this was the whole point of Tailscale; there should be a one click lock to restrict to Tailscale. But the Tailscale documentation is wanting. I actually paid for a candidate for the best firewall front end, it came with "Let us know if we can help!" and radio silence once I explained the problem. Likely, restricting to Tailscale requires a granularity one can only hand-code.

I can write a firewall, I've written plenty in the past, I just couldn't find the several hours to do this as a one-off for me when it should be easy, but I was missing needed information.

Tailscale is justly proud of how it connects machines through uncooperative routers and such. Tailscale SSH should do the same. The idiot's guide to securing a machine so only Tailscale SSH gets through should be to find SSH in the preferences and turn the fucker off.

  ListenAddress 100.x.x.x

It does. You can "turn the fucker off" (as you say) at the OS level and Tailscale SSH will still work. We don't send the Tailscale SSH packets through the OS for it to block them.

Well, Tailscale SSH server support for macOS is still not entirely done. You can build it from source if you're brave (and set an env var to turn it on), but it's not in the product yet by default.

Low latency typing, session resumption etc

The code's at https://github.com/tailscale/tailscale/tree/main/ssh/tailssh for all the details. Which details in particular are you curious about?

In the linked Q&A video with Maisem, you spoke about buying books on Linux to make SSH work. Which books, if you don't mind me asking?

Advanced Programming in the UNIX Environment, 3rd Edition: https://www.amazon.com/gp/product/0321637739

I am not sure if I am confused about something, or maybe there are prod use-cases where the same IDP identity should have different roles/privileges depending on the machine, and Tailscale SSH breaks that?

It'd be really nice just using git transparently and having tailscale take over the git ssh connection and authenticate using taliscale access controls.

At least for personal projects or small teams that'd be quite convenient.

This is safer than OpenSSH.

(OpenSSH, as a piece of software, is extraordinarily safe, and has one of the best records of any memory-unsafe codebases. But OpenSSH as configurable infrastructure is much less safe; people screw it up all the time.)

I think ingress wouldn't be necessary since tailscaled creates a tunnel right?

But how about egress traffic? UDP for WireGuard or something else?

On your host where you're running Tailscale, usually nothing. You can keep everything locked down for ingress. Outbound UDP only, but usually cloud VMs allow outbound traffic already. (This is covered more in https://tailscale.com/kb/1082/firewall-ports/)

Features like Tailscale SSH represent the ruthless removal of annoyances.

edit: grammar

[1]: https://tailscale.com/kb/1193/tailscale-ssh/#ensure-tailscal...

Disclaimer: I am one of the engineers who built Tailscale SSH.

You can secure this by:

- Not enabling the SSH feature on hosts where it's not needed - Creating ACLs so only certain clients are allowed access.

So essentially, just use the same mechanisms as for everything else in Tailscale.