That's good to know, thanks.
My tinc usage was far less demanding. Just a handful of nodes and light traffic, so I didn't experience any issues.
I did migrate to WireGuard about a year ago, and instead of a mesh network, I ended up with a hub-and-spoke configuration[1] which worked fine for my humble needs.
[1]: https://www.procustodibus.com/blog/2020/11/wireguard-hub-and...
Re:Would we still create Nebula today?
by johnmaguire on Saturday October 14, 2023 @06:38PM
(#37884899)
Hey there - sorry for closing your issue! I saw Nate had recommended some commands to run and didn't see a response 8 months later, so I assumed you had moved on.
I've reopened the issue so you can continue to document and look for solutions. :)
Unfortunately, I cannot confirm. Sharing my experience:
I used tinc over multiple years on production servers and it would sometimes create netsplits that did not recover. I also suspect that there's a race or bug in re-keying, which also causes disconnects.
On the netsplit issue, it was me posting alone on the relevant issue [1] over multiple years without response. (I don't expect to get any from free-time maintainers, especially on hard-to-reproduce issues, but it's still important to know that such unsolvable hurdles exist.)
When I switched to Nebula, it improved this situation. But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2].
I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages.
I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss.
Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3].
So I'm currently looking to move the Ceph into WireGuard.
Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break.
[1]: https://github.com/gsliepen/tinc/issues/218
[2]: https://github.com/slackhq/nebula/issues/637
[3]: https://github.com/google/security-research/security/advisor...