Also tested Tail Scale.
Some feedback (only writing this to help you improve):
1. Your UI is horrible. Hire 1-2 front-end/designers and copy everything that tail scale does right.
2. You should add a concept of users with e.g. GitHub as SSO-provider, like tail scale does. Maybe that’s what you’re releasing now?
3. Your docs are very bad compared to tail scale, and you should have much more docs on common scenarios and use cases. For example, mine wasn’t mentioned but is fairly common. You are losing a lot of business here.
4. iOS VPN auto connect functionality is good.
5. You should add some type of global dns, such that we could map all devices/users like this: macbook1.jane.my-company.net (resolve via your network; you host the dns, we provide our domain). Basically what Consul does.
6. If user is authed (see 1 above), auth of devices should be optional.
7. Your language for network rules is too complicated.
8. My impression was that your network software is better than TailScale, but in every other way they beat you (docs, UI, usability, features).
9. iOS app is ugly and have obvious bugs, like you can’t enter text in fields in lower case without hassle.
Couldn’t actually get my use-case to work on tailscale either. They stuff they’re missing is in the works though. Will revisit you both in 6 months.
I’m rooting for you, but you must understand that it’s not only about software, all the packaging around it is also important (and you are severely lacking in this area).
You are 1000% right about UI/UX. It was good for networking software years ago but the ecosystem has generally improved since then.
If anyone who is reading wants to help:
https://jobs.lever.co/zerotier/90436aee-8e55-406d-9053-a0c26...
Location is set to Cincinnati because we want to nucleate more engineers in this region but the position is open to anyone in the USA. We'll hire anywhere if someone is really good. We're remote-first.
Edit:
> 7. Your language for network rules is too complicated.
It's too low-level. We are researching a higher-level way to edit rules in terms of intent rather than the current pf-esque rules language that requires you to deeply understand TCP/IP and such.
That being said it is very powerful and you can do extremely neat stuff with it. It's in some ways more powerful than what you get with enterprise data center SDN products.
I think that's true, but I have basically 0 confidence that I can implement even simple rules using it, let alone anything more complicated.
The thing that was the real show stopper for me and made me switch to Nebula was that there doesn't seem to be a way to self-host a backup controller so that our network can continue to function even if ZeroTier.com is having problems. Unless, that is, I go entirely self-hosted and give up the web management UI, which I think is part of the compelling offer of ZeroTier.
Networks members will continue to be able to communicate with the controller down as long as they were online before the controller went down. Not a full solution, I know.
Otherwise, it's a difficult problem to solve. The only way we could let you run a network controller as a back up right now would be to give you the private key for the controller, which would allow you to change everybody else's network on that controller, too. Not the best of ideas giving that info away!
This just stuck out to me a bit because those seem like two distinct area of expertise
Also I need to edit the part about a BS in CS and add "or demonstrably equivalent skills and experience."
I don't think that's entirely fair to say "horrible". It does everything I've needed of it quite well. The design sensibilities are just... I'm not sure what the right word is, but maybe "ugly" is good enough. I remember when I first looked at it I thought "this is going to be horrible", but once I started using it the functionality was pretty good.
For example, I'd call it better than DefinedNetworks from a usability standpoint, because you aren't clicking in and out of a bunch of things, but DN is definitely easier on the eyes.
Until 2 weeks ago when my Windows machines absolutely stopped seeing each other and communicating with each other.
I made a post on ZeroTier discussion group 2 weeks ago with zero replies so far: https://discuss.zerotier.com/t/windows-machines-lost-access-...
If i can get some help - would be amazing.
On both computers do: Network connections->Right click ZT network->Properties->Select (TCP/IPv4), probs 5th down. ->Properties->Advanced->Untick automatic metric-> set interface metric value to 1. Potentially reboot, though I've not needed to before.
Just for some visuals if needed https://support.parsec.app/hc/en-us/articles/115002766652-Us...
have you worked through: https://docs.zerotier.com/zerotier/troubleshooting/ ?
However suddenly one laptop was able to connect to 2 other windows boxes (even though other Windows machines cant).
This tells me it's not ZT fault, but rather everlasting update and restriction nonsense from windows is a culprit
Keep up the great work!
(1) We use AES for performance and other reasons. ChaCha is much faster than AES in software but AES blows it away by 2-4X in hardware. We have customers using ZeroTier in an SD-WAN setting who are leveraging AES acceleration (with some of their own custom code) in network accelerators and other devices. We also think we can beat both in-kernel WG and ipsec in V2 on Linux using io_uring because the cryptographic part of ZeroTier is 2-4X faster than WG on "real hardware." My M1 Mac does 2GiB/sec/core and newer Intel/AMD chips with AVX VAES can do as much as 10GiB/sec/core.
(2) We have a bunch of customers who want FIPS. Yes, we know. We're not big fans either but they're large customers and whole industries. We also have a pretty good no-compromise plan for this that treats FIPS as a floor for cryptographic quality not a ceiling and doesn't add meaningful complexity. This is a V2 thing. (You can bootstrap WG with keys from a NIST ECC key exchange to get the asymmetric part of FIPS, but ChaCha is not FIPS.)
(3) We want to maintain backward compatibility with millions and millions of ZeroTier devices out there, at least as far back as we can.
(4) ZeroTier uses a construction where the identity is the outer envelope and then (in V2) the ephemeral forward secrecy keys are the inner envelope. Wireguard does it the other way around. Both are equivalent in terms of cryptographic strength (and I think in noise this is a choice, but it's been a while since I read through that). The advantage of the WG construction is that session participants can't be identified from packets. That's a disadvantage in enterprise though, because they want that. It lets us implement elegant solutions to making a mesh P2P network play nice with enterprise firewall and internal/DMZ isolation practices among other things. It also lets our roots/relays basically do something analogous to MPLS, pumping vast amounts of traffic with virtually no CPU overhead because they don't need to decrypt anything.
We toyed around with making an AES fork of Wireguard, basically use the exact same construction as WG with AES-GMAC-SIV instead of ChaChaPoly. Would be totally fine cryptographically but see #4.
We also support virtualization at layer 2 rather than just layer 3, which is important for a bunch of our industrial use cases. It would be possible though to stuff L2 into WG pretty easily, so that's not really a reason. WG is just a transport protocol so you could stuff anything into it. Fly.io used it as a light alternative to TLS I think. It's just normally used to carry L3 IP.
Edit: also in V2 we are modularizing things, separating the transport protocol from the rest of the code. It'll be in open source Rust and very portable. In our (currently private) Rust repo vl1/ is only about 5K lines of Rust including tons of comments and non-crypto stuff. We also have plans to document it a lot better.
Does this mean that any node along the network path would be able to track communications between nodes? Maybe a snoop wouldn't be able to pin an envelope to a named identity, but they would be able to track which other identities its connecting to down to the communication bandwidth between each source and destination and correlate the identities of those connections between separate devices. Is this an accurate description?
I have some thoughts about features to really defeat this. We've got an entry on our feature queue for guard nodes you can run in the cloud and have also researched implementing a form of "light" onion routing. It wouldn't be intended to be as strong (or as slow) as Tor, but sufficient to prevent passive dragnet surveillance.
In any case there is a conflict here between goals. So far the performance, rapid startup, and enterprise goals have won out over the small privacy benefit of the ephemeral-outer-key construction. It's a scope thing. So far we've never scoped ZeroTier as a meta-data-concealing protocol like Tor, just as a P2P mesh protocol for easy remote access between devices and locations.
Still... it's tempting to try to find something in the V2 design to at least mitigate this issue. It's something we're thinking about.
I think a TL;DR on my reply above:
I think the difference in privacy between identity(ephemeral(x)) and ephemeral(identity(x)) is not significant. It's more honest to tell people that our product offers only content privacy not meta-data privacy unless we are truly implementing strong countermeasures against tracking by ISPs or others that can pervasively sniff the net. Just nixing the identity key visibility is not a strong countermeasure for the reasons I described.
We've been hard at work on this for a very long time, and we're obviously quite happy today! This is a must-have or really-want for a huge number of business customers.
That being said, we don't and are not going to require SSO sign-in. You'll always be able to just authorize a device. (On SSO networks that's done by setting them to SSO-exempt.) You can also self-host network controllers like always, and we're planning on making that easier in the future not harder. We're all about not forcing you into ecosystems (SSO can do that) and about decentralization.
As for other stuff coming in the near-mid future: version 2.0 is not dead. We obviously got massively derailed (mostly good but very distracting things) and under-estimated the scope and time, but it's still moving forward. We have not announced this elsewhere yet, but it's a near-total port to Rust. This is to get into a more modern language but also to use a safe language for security reasons. This has added time to the job but we're of the opinion that security-related software not written in a safe language is going to be considered a bad thing in the near future (if not already).
V2 also has some significant cryptography improvements that bring us more to parity with more modern constructions like the noise framework. This part has been going slow too because we've been moving carefully and soliciting a lot of peer review both informal and formally hired. Cryptography isn't something you just toss out the door and YOLO. :)
Last but not least we are planning on some kind of transition from the BSL back to an OSI-compliant licensing scheme, but want to think this through rather than flail around.
People at this site have really been fans and have helped us a lot over the years, and we're grateful. Thanks!