I disagree - spent a considerable amount of time with zerotier as a possible replacement of a small sized ipsec mesh (4 sites) and it failed horribly. Had commercial support, different hardware and even virtualized it. Latency was a major issue and quality of the links were erratic to say the least. Don't get me wrong, I think zerotier is great, but it's not prime time.

I've had a similar experience. In particular links will just "drop out" for periods of time. The public forwarding nodes were overburdened for quite a while. I set up my own "moon", but one of the sites has a cranky NAT, which will let a connection through for a while, then fail. It seems to take at least 30 seconds for zerotier to "notice" this and switch back to forwarding via the moon. Maybe the new multipath will help?

How is the VPN responsible for your crappy underlay network?

Rather obviously it isn't. I'm not sure why you'd even ask.

I'm not the only one with external NAT that I can't do anything about; the question is what to do to mitigate this.

Switching to an explicit hub-and-spoke model would work around this, but at the expense of what I consider one of ZeroTier's biggest strengths: transparent meshing. If two machines in the network are on the same LAN, I'd like them to use that rather than the network.

Faster detection of the failure of the NAT-piercing peer-to-peer link, with fallback to the "moon" while the peer-to-peer link is being re-established, would substantially increase the usability for people, like me, who are stuck with the NAT they've got. As I alluded to, the new multi-path features that ZeroTier is getting might help with that.

Your physical network settings likely didn't allow direct connections between peers. Fix that and it'll work fine.

If it’s replacing an ipsec mesh that’s pretty hard to believe. And if that was the issue and commercial support couldn’t even identify that as the cause, ZeroTier has bigger issues.

If all sites are behind symmetric NATs, there's not much ZeroTier could do to help aside from telling him to assign direct mappings on the NAT/Firewall to each ZT instance. Symmetric NATs are antithetical to peer to peer communication. Many I've run across in the wild have special rules to handle IPSec which won't exist for other lesser known protocols. It's also possible the user wasn't willing or able to make network configuration changes to make those p2p connections possible. Without seeing what the user tried & support recommended, it's not really fair to throw out such baseless accusations.

ZeroTier uses UDP. That's hardly "lesser known" than IPSec.

"lesser known" as in protocols such as IPSec, ZeroTier, WireGuard, etc. Of which IPSec has been around forever and many NATs/Firewalls have special handling rules built in, just as @api mentioned in another comment. Yes, ZeroTier uses UDP underneath, but that doesn't mean symmetric NATs don't/won't cause havoc to peer to peer protocols using UDP.

Wrong layers of the network. IPSec is comparable to TCP/UDP, not wireguard/zerotier. It’s L4 and NAT can’t have enough intelligence to setup IPSec meshes without explicit configuration.

Finally, how can ZeroTier’s support be so incompetent to not recognize connectivity issues between endpoints? That’s one of the few things that goes wrong with tunnel meshes.

It was probably behind finicky and heavily restrictive symmetric NAT (very p2p-hostile) but with IPSec ALG in the NAT, making it work fine with IPSec but horribly with anything else. This is common in "enterprise" settings and hard to diagnose without direct remote access to run NAT characterization tests.

Symmetric NAT basically breaks everything that doesn't use a simple client/server hub-and-spoke networking model.

Yeah, i hear about that regularly but didn't look into it. I must say i'm not really happy about the whole business thing. The arch wiki says you need an account, i'm not sure if that is true but if it is, it's a non-starter for me. If you have good technical refs to prove me wrong i'd be happy to hear.

You only need an account if you're using ZeroTier's hosted network controller service. You can run your own controller as well. The code to do so is in every instance of ZeroTier. https://github.com/zerotier/ZeroTierOne/tree/master/controll...

Agreed. That's my daily driver. No pun intended.