rhuber

These models are basically a compression of everything we already made. Our books, our code, every stack overflow answer, the dumb forum arguments we had at 2am. The companies creating these models didn't create that knowledge, they scraped it and compressed it, and what comes back out is mostly our own collective output re-encoded and handed back to us. The work is expensive and valuable, so I don't mean to diminish that, but exclusivity on this seems completely unfair. These organizations already play fast-and-loose with copyright, but even if they didn't, we can't all go retroactively copyright every word we've posted on the internet.

We all know how patents work. They are a controversial topic, especially in computing, but their intent is to disincentivize secrecy. The compute needed to create these models cost exorbitant sums of money, and I'm all for these companies profiting from this investment, and their own hard work. The rate at which models are being replaced is already staggering, and the competitive advantage of a model from 12 months ago is generally nonexistent. I can't imagine a good argument for Opus 3 or GPT 4 not being open weight today. Their economic value is limited.

Importantly, I'm not proposing that these companies give us a blueprint to recreate their models at all. The proprietary systems they use to create these things should stay their competitive advantage for as long as they wish.

I truly believe these companies will dominate knowledge work indefinitely, and in this case, a startup is not a viable path to competition. You need hundreds of millions of dollars to even get started. This is a game for the already rich. But if you want to make their use of our entire corpus of online information fairer, they shouldn't have exclusivity on the output indefinitely.

That's totally reasonable, and I agree that using something hosted entirely by a 3rd party makes sense for some use cases. Our reason goes a bit beyond security conecerns, in this case. We built Nebula for large scale deployments, and because of that, we have made decisions that lean into that model for hosting.

Our decision to leave lighthouse hosting in the hands of users has one primary rationale: We want users to have complete control their network availability. Any downtime of our service should not impact their network availability. You can even host some of your lighthouses inside of network boundaries to ensure that an internal network functions properly if its connection to the internet is interrupted. Some software will continue to work, but new connections are often not possible, and the network can degrade rapidly.

Relays are are a similar story, but with an additional reason: We don't have to limit our customers' relay bandwidth due to cost. When hosting relays on behalf of others, we would be transiting a lot of traffic, which has an associated (sometimes unpredictable) cost. By letting our customers host relays, they can ensure relay traffic is just as fast as direclt connections.

Thanks Harrison, hope you're well!

That's a great question! One of the things I enjoyed during my time at Slack was their willingness to contribute to open source projects. We had similar IP clauses, but asking permission to open source things was straightforward.

The most important concern (IMO), was considering whether we could commit to properly maintaining a project. Before open sourcing anything, you need to discuss how you'll go about managing an issue and pull request backlog, so that people don't come across "dead" projects under your stewardship.

In a high growth startup, I do think something like this could happen again, but as a company grows, there are certainly more layers that can make it difficult to share things openly.

(*blog post author here)

Thanks for sharing this on HN! I'll keep an eye on the comments and try to answer questions that come up.

We need to do a better job of this and I'm really sorry you had a not-great experience with expiration. Totally agree with your take.

We have an automated set of ansible scripts that spin up large groups of hosts for Nebula performance regression testing, and a while back I added zerotier, tailscale, wireguard-userspace, wireguard, tinc, ipsec, and openvpn to that automation so I could get a sense of where things stand. I spent a lot of time optimizing each of the above options to make fair comparisons, but it was mostly for mine and the team's curiosity, and we weren't interested in playing benchmark-fight with similar softwares of the world.

Publishing repeatable benchmarks is hard, and when doing open source work, it just hasn't been a priority. As I replied above, if I'm going to say it I should prove it, and I promised to do just that.

And a counterpoint: tailscale does mention in the "Tailscale vs Nebula" article on their website that performance is just about the same but similarly provides no proof. This is motivation enough for me to show proof of the opposite, I guess.

I mean... the title of the Tailscale blog post is "Tailscale raises $100M… to fix the Internet", and that's pretty massive scale. /s

I don't have 100k hosts on a large network to test deploying Tailscale, but if I did, I'd be benchmarking the cpu/network/storage overhead of telling 99,999 hosts about a new one that comes online, every time that happens, or every time its pubkey changes. You can optimize this away _if_ your "fan out" is not as large, but there are plenty of cases where every host on your network needs to talk to a particular host, so all of them need to know about its keys as soon as possible.

Again these aren't unsolvable problems, to a point, but we didn't want to solve a problem when we could avoid it entirely, so that's the path we chose. It removes complexity and is a good part of the reason the system we built has been resilient.

A complaint some people express about tailscale is the battery life on mobile (or at least iOS). This exists because there is coordination overhead on even idle tailscale nodes. Back when we ported Nebula to iOS, we sweated details like "how often it wakes the radios" and did a lot of profiling. I never turn Nebula "off" on my iPhone, and it just sits in there in the background not using any resources most of the time.

We worked hard to optimize this out of our architecture, so that Nebula avoids generating traffic that is unrelated to the actual communication between hosts or lookups to lighthouses. An idle nebula tunnel can truly be idle indefinitely, and that also matters as the set of hosts becomes larger.

I do not think the Nebula project and Tailscale are direct replacements for each other in any fashion, and afaik neither is trying to be. I'm just pointing out that different design goals led to unique advantages and disadvantages to each architecture.

The Nebula CA we built at Slack was very specific to Slack's internal devops, and just wasn't generalizable. It is highly automated there, and is custom tooling, just as you describe. The open source version is somewhat bare bones (a command line tool for CA vs something like vault).

I will say that the OSS tooling of Nebula is everything someone needs to stand up an entire working network on every common platform (linux/mac/windows/ios/android), but there is a definite gap in simplification that we need to address to make it easier for smaller scale use cases.

We actually have a managed enterprise Nebula offering at my current gig, but that's rather a different market than Tailscale, so I'm avoiding talking as that company as opposed to a Nebula OSS project lead. The commercial offering is targeted at large enterprises, because that's the market where Nebula has unique advantages. It also means we don't currently have a freemium or smb type offering, and are not prioritizing creating one at all. I don't want to give people false hope that we will, and would prefer to see the OSS project improve to address the small-medium use cases.

Another fair criticism. We will publish the benchmarks and make them repeatable (which most existing ones I've found don't bother to do). We hadn't done so because Tailscale isn't really seen as a direct competitor to what the Nebula project is doing, but if people want numbers, that's a thing we are happy to provide.

Fair enough. I am sure the key distribution is fast and all that, but not needing peer key distribution at all was a goal and the overhead associated is less scalable than just not doing it at all. Regardless, very cool that you can handle that many nodes, which is a hard problem. I assume you do just-in-time key distribution or something, because (n-1) distribution of peer keys would be ... less than ideal.

Anywho, the more important bit is my point about performance. Nebula is significantly faster than userspace Wireguard, and plain userspace Wireguard is (last I checked) a bit faster than Tailscale, due to the additional code needed for things like your ACLs. At gigabit type scale it is probably fine and not noticeable, but at Slack, we needed to scale to 10G+ on links, while ensuring we didn't take a significant hit on CPU resources.

Again, I think Tailscale is very good for its target use case as a VPN replacement, and congrats on raising these funds!

It does! In fact replacing AWS security groups and making them cross region and cross platform was probably the first goal of the project. My coauthor, Nate, wrote Nebula's internal firewall code before we wrote a single line of the actual protocol, because he wanted to ensure it was performant enough for massive scale.

(Nebula coauthor here)

People sometimes ask me to describe the differences between Nebula and Tailscale. One of the most important relates to performance and scale. Nebula can handle the amount of internal network traffic and scalability of nodes (100k+ nodes, constant churn) required on a large network like Slack's, but Tailscale cannot. Tailscale's performance is fine for many situations, but not suitable for infrastructure. It is just a fundamentally different set of goals.

Nebula was created and open sourced before Tailscale was offering their product, but their architecture is similar to older offerings in the market, and is something we purposely avoided when creating Nebula.

Fwiw, I even recommend Tailscale to friends who want to do things like connect to their Plex server or Synology or [other thing] at home remotely. It simplifies this kind of thing greatly and doesn't require you to set up any infrastructure you control directly, which can be a headache for folks who just want to reach a handful of computers/devices.

(coauthor of Nebula)

We briefly considered building something atop Wireguard in the early days of Nebula, but decided not to do so because of scaling. Wireguard's protocol necessitates that all nodes have existing keypairs for each other ahead of time. At Slack's scale, that means every time a fresh node is launched, you would have to tell 50,000 other nodes it exists.

Obviously you can smarten this up and tell only hosts it might talk to. But this adds complexity. Using PKI eliminates this key distribution problem and means that you don't have the same scaling limitations as something built on WG.

Wireguard is a very very good VPN, but I cannot imagine trying to run something on the scale of tens of thousands of nodes when you need such complex coordination systems to exchange keys/trust, especially in a dynamic environment where nodes are coming and going all the time.

I totally get that it solvable overall, but Slack has had 4 years of nearly perfect uptime on Nebula, whilst using it to pass >95% of all backend traffic. These considerations may seem simple to address, but there are fundamentals that mattered and led us to writing Nebula. We didn't want to create something new, but to do what Slack needed, we had to.

Actually it does do that, you can trust multiple CAs in a single instance and even write firewalls scoped to CAs.

"Hey guys, remember when freenode was taken over by one guy?" (It was, like, yesterday.)

It's hard to overstate the dangers of over-centralisation like this, and I say it as a person who used freenode professionally.

https://www.kline.sh/

I wasn't commenting on the strength of RSA-1024, per se, but on the assumed age of that key. OpenSSH's ssh-keygen hasn't defaulted to 1024 bit RSA keys since before version 4.2, in 2005. (I had to look it up: https://www.openssh.com/releasenotes.html)

You can still generate a 1024 bit RSA key, but someone would have to go out of their way to do so, and I can't imagine why they would have done that in the past .. decade?

The author of this article should consider following their own advice, since they have a woefully outdated RSA-1024 ssh key securing their GitHub account.

$ curl -s https://github.com/apenwarr.keys > blah

$ ssh-keygen -l -f blah

1024 SHA256:1IWAUSXOcCKLcmOdAec8JbDt3T75udA4KSpRosEWUaU no comment (RSA)

(update: they have now replaced it with an RSA 2048 bit key. progress.)

It is, admittedly, impossible for me to be unbiased in this discussion (coauthor of Nebula, hi), but I strongly disagree that your client code is your most important component, from a trust perspective.

Your coordination server tells every node about every other node and distributes the keys for the entire network. Everything on a tailscale network implicitly trusts your coordination service.

If an individual client is compromised, code or otherwise, the effect is more limited than your coordination service being compromised, in which case the entire system's trust is broken.

Every aspiring programmer should learn C. They should also learn multiple dialects of ASM. Learning these things helps you better understand how computers fundamentally work, and knowing how computers work pushes you to write better code in any language. Ignoring how computers organize data can lead to inefficient code in any language.

That said, if you are writing something new, you should carefully consider whether C is the best choice before using it. If you are working within a nearly-impossible-to-replace + enormous codebase (such as the Linux kernel), it is the only option. If your project's #1 goal is performance above all else, and you are a seasoned or aspiring expert, perhaps C is the best choice. The majority of people writing software do not fall into either of these categories.

Hi - Ryan from the blog post here.

eBPF is great, and we plan to support it as our collection mechanism, but not today and maybe not soon. When we wrote go-audit, there were few, (if any?), distros that had eBPF kernel support. Now Ubuntu 16.04 has support, as do many others, and that is great!

But..

Releasing a tool that will work for most people today is very different to releasing one that will work for most people in a couple of years. We are sharing this imperfect tool that uses an using an imperfect syscall monitoring mechanism, both of which were written by imperfect humans.

We have found plenty of problems with auditd during development (some real facepalm stuff, to be sure), but it also does many useful things. We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot.

If you don't trust auditd, that's a totally valid opinion to hold. I welcome criticism of our approach, but a PSA that doesn't involve having testing our tool is rather unfair.

There are a few reasons for my comment about going against the spirit:

1) https://hackerone.com/disclosure-guidelines states:

"If 180 days have elapsed with the Response Team being unable or unwilling to provide a disclosure timeline, the contents of the Bug Report may be publicly disclosed by the Researcher. We believe transparency is in the public's best interest in these extreme cases."

2) He set an arbitrary 90 day disclosure checkpoint.

3) We explicitly asked for more time in dealing with the bug.

4) We had an extremely negative experience with him during his first report. He was unnecessarily adversarial when we patiently explained that he had not found a vulnerability.

---------

Within the HackerOne interface, a "Duplicate" is actually listed as a Closed:Duplicate issue, and doesn't appear in the Open issues tab at all. Perhaps a method of attaching duplicates to the original and allowing communication between all involved is useful? ¯\_(ツ)_/¯

Notes on “a little note”.

Hi, this is Ryan. I work at Slack.

Bug bounties are great, but managing them can be a challenge. Like many companies that run a popular bounty program, we receive quite a few vague reports, invalid reports, and reports generated by automated scanners. We work through these daily to ensure we are focused on the bugs that can have an adverse impact on our users.

We have positive interactions with the people who report bugs, and we appreciate the hard work involved in uncovering issues. If you find a bug, report it via HackerOne and we will reward your work. We have rewarded researchers for over 300 bugs found so far!

Anshuman sent us the first report in December. At a glance his report appeared to be well written and detailed. When triaging bugs, those two things are especially helpful. (We appreciate well written POCs!) We reproduce every report received, so below I will convert his report into a description of the problem and a series of steps needed to reproduce it (original report quoted).

------------

From the report:

“Slack users are allowed to share files (posts, snippets) with other users and within channels.”

True

“When a file is shared in a channel and unshared again, it is clearly mentioned on the website that: Un-sharing the file will not remove existing share and comment messages, but it will keep any future comments from appearing in the channel.”

True. This is what the un-sharing feature does. As stated above, files.unshare is in no way an access control feature.

“This makes it obvious that on sharing and then unsharing a file within a channel, it will still remain shared and can be viewed by others on that channel. This is the way it is supposed to be.”

True. Again, this API call is used to stop new comments about a file from appearing in a channel, not to remove the file from a channel. (Deleting is done by the files.delete method) So far no bug, just things working as expected.

“Now, when a file is shared with a Slack user, currently, there is no way to unshare it again from the UI.”

True.

“But, this can be easily done by sending a request to the https://<domain>.slack.com/api/files.unshare end point instead of the https://<domain>.slack.com/api/files.share end point.”

The reporter is proposing that the victim call files.unshare to utilize a “hidden feature”. The reason a user might do this is left to the imagination.

“It is as simple as that.”

There is no instance of files.unshare being called this way in the UI, because that is not what it does. Calling an API method that is not documented is never guaranteed to do what you assume it does.

------------

What Anshuman has created is a scenario where the “victim” must:

1) Use Slack via the Web, Mobile or Desktop Application. 2) Share a file with another user 3) Observe API calls (or read the javascript). 4) Make an assumption about what api/files.unshare is used for. 5) Call that API method directly. (curl, js, whatever..). 6) Expect that the method does what you have guessed. (it doesn’t, because the reporter's guess was incorrect.)

------------

Testing this report involved working with multiple developers to review the nature of files.share and what the impact of this bug would be. At the end of our investigation we replied to the reporter saying that we appreciate his effort, but this is not a vulnerability, because files.unshare is never used in this way. Unfortunately, we then received this message from Anshuman:

“I am giving you a heads up that I will be blogging about this sometime today. Thanks for your time.”

So after hours spent reproducing this and then explaining to Anshuman why it isn't a vulnerability, his reaction was to create a blog post titled “Hidden Feature in Slack leads to Unauthorized Information Leakage of Files”.

I believe that HackerOne is a valuable platform, and outside of this instance our experience has been extremely positive. We will continue to use it and look forward to working with new people.

Btw, I’m not off the hook, because I did something wrong too. I failed to keep Anshuman updated on a second report he filed in December. I absolutely agree that bug bounty participants should receive timely replies to their queries. This oversight is regrettable and this mistake will not be made again. My apologies to Anshuman for not keeping him updated on the status of the bug, which would have allowed proper coordination and disclosure.

Good Hunting,

Ryan