Edit: I popped this up because to me is absolutely miserable from a big company to just say: "Thanks, but we were aware of this".
https://docs.github.com/en/pull-requests/collaborating-with-...
I don't think they're being underhanded exactly... they're just making a terrible decision. Quoting from the article:
> The average user views the separation of private and public repositories as a security boundary, and understandably believes that any data located in a private repository cannot be accessed by public users. Unfortunately, as we documented above, that is not always true. Whatsmore, the act of deletion implies the destruction of data. As we saw above, deleting a repository or fork does not mean your commit data is actually deleted.
He found a security problem, they decided not to act on it, but it was still an acknowledged security problem
It's only a bug if it's not intended
In this case the question is moot, as this doesn't involve remote code execution.
If the criteria for bug is "not intended", and that's solely judged by the company, then broken auth et al. suddenly become part of their product design.
If it quacks like a bug, it's a bug.
If the (class of) problem is already known, it’s not worth rewarding.
At the end it boils down to: is Github being honest and fair in answering the bug bounty reports?
If you think it is, cool.
If you don't, maybe it's not worth playing ball with Github's bug bounty process
The subtext of this thread is that companies should reward any research that turns up surprising or user-hostile behavior in products. It's good to want things. But that is not the point of a security bug bounty.
So much this. It's pretty clear that most people commenting on this thread have never been involved in a bug bounty program on the company's side.
Bug bounty programs get a lot of reports, most of which are frankly useless and many of which are cases of intended behavior subjectively perceived as problematic. Sifting through that mess is a lot of work, and if you regularly pay out on unhelpful reports you end up with many more unhelpful reports.
This particular case definitely feels like one where the intended behavior is horribly broken, but there are absolutely many cases where "this is intended" is the only valid answer to a report.
Companies can join/setup a bug bounty program, and just use it as a fig leaf for pretending to care about their own product/service's security.
Of course bug bounties can and are abused daily by people who report trivial non-issues in the hope of compensation
But in the same way, companies can also be bad actors in the way that they engage with bounties. I would usually expect big names (like Google, Apple, Github, etc.) to be trustworthy...
[*] Of course what stops companies is precisely them not being seen as trustworthy actors in the bug bounty system anymore... And for now, that's a decision that individuals have to make themselves
If you don't trust the company running a bounty, don't participate. There are more lucrative ways to put vulnerability research skill to use.
Calling this a bug is like reporting that telnet sends information unencrypted.
The actual bug is in the way that their UX paradigm sets user expectations.
Github chooses to store all "Github forks" in the same repository, and allow accessing things in that repository even when they are not reachable by the refs in the namespace of one "fork". That is purely a Github decision.
"We surveyed a few (literally 3) commonly-forked public repositories from a large AI company and easily found 40 valid API keys from deleted forks."
This is how your customers get their entire cloud taken over, because you made a stupid, stupid decision and instead of fixing it when warned (repeatedly!) you instead decide to just blame the customer for not reading page 537 paragraph 3 subsection B about the counter-intuitive security footgun you've left in your product.
This is negligence, pure and simple.
The thing about exposing commits that were only ever in a private repo is pretty indefensible, but not garbage collecting public commits on delete shouldn't matter.
Why would anyone think that a private fork is "published"!?
This is the footgun here: The UI is telling you that nobody can see the secrets you committed to your private copy, but actually it is widely accessible.
A similar example of UI-vs-reality mismatch that I've noticed recently is the Azure Store Account "public" visibility. By default, it uses your authenticated account for RBAC access checks, so if you click around it'll say something like "you don't have browse access". This looks secure, but attempting access anonymously would have succeeded!
I had a customer recently where this happened -- they clicked through every Storage Account to "check" them, convinced themselves they were secure, meanwhile they had database backups with PII accessible to world+dog!
1. Public "forks" are just namespaced branches that share an underlying repo
2. They don't run the garbage collector all the time
I'd be surprised if those weren't true.
Like I said, the behavior with private forks sounds indefensible.
The OP is mixing together multiple things. Being able to access deleted public data isn't that surprising and definitely isn't a security issue as far as leaking keys is concerned (it was already public. Assume it has been cloned). Being able to access private forks is a footgun/issue. They should be garbage collecting as part of public repo creation so that unreferenced commits from private forks aren't included.
I do see this regularly in my work. All but one dev team I’ve worked with over the last few years has done this.
Anyone who put sensitive content in a git repo should consider published anyway. Git is a decentralized tool, as a company you cannot control the amount of git remotes that may host your code. Considering your code is only hosted as a private repo in a specific remote git server is at best naive. This is without even considering the amount of copies that are stored in dev computers.
Besides, anyone who put stuff on a third party publicly accessible infrastructure should consider it published anyway as breaches happen all the time.
If you happen to have api keys stored in a git repo, the only viable response is rotating those keys.
My employer doesn't pay out for known security issues, especially if we have mitigating controls.
A lot of people spam us with vulnerability reports from security tools we already use. At least half of them turn out to be false positives we are already aware of. In my opinion, running a bug bounty program at all is a net negative for us. We aren't large enough to get the attention of anyone competent.
Lets get back to what we had before when multiple people can find the same bug and exploit if needed. Now we have the one person who finds the bug it gets patched and they don't get paid.
I'm honestly not yet convinced that is enough here - I've fallen victim to this without realizing it - the behaviour here is so far removed from how I suspect most user's mental model of github.com works. For me none of the exposed data is sensitive, but the point remains I was totally unawares it would be retrievable like this.
If the behaviour flies so against the grain, just publishing it in a help doc is not enough I'd argue. The linked article makes the exact same argument:
> "The average user views the separation of private and public repositories as a security boundary, and understandably believes that any data located in a private repository cannot be accessed by public users. Unfortunately, as we documented above, that is not always true. Whatsmore, the act of deletion implies the destruction of data. As we saw above, deleting a repository or fork does not mean your commit data is actually deleted."
There's a deeper problem here, which is that making the UX on hosting sites less surprising doesn't fix the underlying problem. There is a best-practices response to commiting hazmat to a repository: revoke the hazmat, so that its disclosure no long matters. You have to do this anyways. If you can't, you should be in contact with Github directly to remove it.
Also, you can revoke an API key, but you can't revoke a company-proprietary algorithm that you implemented into a fork of a public project.
The solution for that is better UX.
I supposed the branding and UI for it could be a counter argument, but then again Github allows regular branch creation / committing / merging in their UI. Their main value add (not downplaying it—it's huge) on top of git (besides ancillary things like CI / linters) is the ability to comment on a branch's diff, i.e. a PR Review.
Regardless, the vulnerability in Github forks falls squarely on Github, and is not mitigated by Git being hard to understand in the first place.
1. GitHub has a nasty privacy/security hole, where commonsense expectations about the meanings of common words are violated by the system.
2. Github has publicly announced that they don't care about this part of user data security (private code), so won't pay people to know tell them what they alreay know and announced.
Github won't pay you to tell them they are wrong when everyone alreay knows.
Beyond that is is also a batshit crazy implementation. Just I imagine AWS would still allow AWS credentials to give access to a deleted account
"Anyone can access deleted and private repository data on GitHub"
But if you buy someone non-refundable tickets to a concert they already have tickets for, you aren't owed compensation.
In this case, the spot turned out to be a freckle, which everyone involved already knew was a freckle (since it was documented), and if anyone owes anyone anything, it's the researcher that owes github for wasting their time.
Exactly.
> The receiver is obligated to return the favor.
Not at all. This is a very toxic expectation.
For companies like Microsoft, which a CSRB audit showed that their security culture 'inadequate', the risk of disclosure with a POC is about the only tool we have to enforce their side of the Shared Responsibility Model.
Even the largest IT spender in the world, the US government has moved more from the carrot to the stick model. If they have to do it so do we.
Unfortunately as publishing a 'bad practices' list by us doesn't invoke the risk of EULA busting gross negligence claims, responsible disclosure is one of the few tools we have.
The thing to keep in mind is that large-scale bug bounty programs make their own incentive weather. People game the hell out of them. If you ack and fix sev:info bugs, people submit lots more sev:info bugs, and now your security program has been reoriented around the dumbest bugs --- the opposite of what you want a bounty program to do.
To provide one example I reported to a large cloud provider that their managed Kubernetes system exposed the Insecure port to the container network, meaning that anyone with access to one container automatically got cluster-admin rights. That pretty clearly seems like not a good security choice, but probably hard to fix if they were relying on that behaviour (which I'm guessing they were).
Their response was to say it was a "best practice" behaviour (no bounty applicable) and that they'd look to fix and asked me not to publicly mention it. Then they deprecated the entire product 6 months later :D
That's one example but I've seen similar behaviour multiple times for things that are more architecture choices than direct bugs, which makes me think reporting such things isn't always welcome by the program owners.
You would prefer this company to have made a different strategic choice about what to spend engineering time on, and that's fine. But engineering cycles are finite, so whatever time they'd spend configuring K8s differently is time they wouldn't be spending on some other security goal, which, for all we know, was more important. Software is fathomlessly awful, after all.
That doesn't mean the report isn't legitimate (being cluster-admin with no authentication is generally considered not to be a good security idea, in fact it's about as bad as it could get without putting the insecure port on the Internet), but that bug bounties aren't architected to accept that kind of issue. The challenge with this is it means that bug bounty researchers won't look for that kind of (legitimate) security issue as they get to know the programs often won't pay out.
Personally, I don't ever report bug bounties for the money reward, but so I don't get shouted at by companies when I write a blog or do a talk that covers the issues :) In this case I was a bit annoyed that they combined telling me it wasn't a bug, with asking me not to mention it publicly for 6+ months (IIRC they credited me like years after the fact).
Sounds sus. If they said it's "best practice" then certainly they'd want you to tell everyone about it.
If a bug bounty is denied I'd assume you're free to talk about it publicly.
GitHub clearly knew. Would you prefer that a vendor lie?
The security implications are also minor. The only problem really is with making a fork of a private repo public - that should only make what exists in that fork public and not any other objects. Something that was already public staying public even when you delete it from your repo is not a security issue at all. Keys you have ever been pushed to a public repo should be revoked no matter what, with or without this GitGub feature.
This causes lots of problems for repositories that are private with the expectation that companies will make private forks with their own private changes.
Someone once pushed a bunch of console SDKs (under strict NDA) to a private fork without knowing this. Now that code is just there, if you can guess the commit hash, forever. Literally nothing can be done to remove it. Great.
* You add someone to your private repo.
* After some time, you revoke their access.
As long as they keep a fork (which you can't control) they can use this same method to access new commits on the repo and commits from other private forks.
Back in 2018, this was a resolved as won't fix, but it also wasn't documented.
In my experience, there was two reasons for this behavior:
1. They don't want to spin dev cycles on something that isn't directly related to revenue (e.g. security) 2. Developers don't have the same mindset as someone who's whole job is security. So they think something is fine when it's really not.
Is my repository accessible?
As far as I know, it is not accessible.
What I don't know is if in 3 months you DO set your remote origin to that fork to for instance, pull upstream patches into your private repo, you're still not pushing, only pulling, so I would THINK they'd still never get your changes, but I don't know if git does some sort of log sync when you do a pull as well.
Maybe that would wind up having the commit hash available.
the issue is the `fork` mechanism of github is not semantically like a `git clone`
it's more like creating a larger git repo in which all forks weather private or not are contained and which doesn't properly implement access management (at least point 2&3 wouldn't be an issue if they did)
there are also some implications form point 1 that forks do in some way infer with gc-ing orphan commits (e.g. the non synced commits in he deleted repo in point 1) at least that should be a bug IMHO one which also costs them storage
(also to be clear for me 2&3 are security vulnerabilities no matter if they are classified as intended behavior)
obviously not a fan of this policy though
> A fork is a new repository that shares code and visibility settings with the upstream repository. All forks of public repositories are public. You cannot change the visibility of a fork.
[1] https://docs.github.com/en/pull-requests/collaborating-with-...
After some changes which include deleting sensitive information and proprietary code, and squashing all the history to one commit, they change the repo to public.
According to this article, any commit on either repo which was made before the 2nd repo was made public, can still be accessed on the public repo.
I know this might look like a valid approach on the first glance but... it is stupid for anyone who knows how git or GitHub API works? Remote (GitHub's) reflog is not GC'd immediately, you can try to get commit hashes from events history via API, and then try to get commits from reflog.
Do not squash your commits and make the repository public. Instead, make a new repository and add the code there.
You need to know how git works and GitHub's API. I would say I have a pretty good understanding about how (local) git works internally, but was deeply surprised about GitHub's brute-forceable short commit IDs and the existence of a public log of all reflog activity [1].
When the article said "You might think you’re protected by needing to know the commit hash. You’re not. The hash is discoverable. More on that later." I was not able to deduce what would come later. Meanwhile, data access by hash seemed like a non-issue to me – how would you compute the hash without having the data in the first place? Checking that a certain file exists in a private branch might be an information disclosure, but gi not usually problematic.
And in any case, GitHub has grown so far away from its roots as a simple git hoster that implicit expectations change as well. If I self-host my git repository, my mental model is very close to git internals. If I use GitHub's web interface to click myself a repository with complex access rights, I assume they have concepts in place to thoroughly enforce these access rights. I mean, GitHub organizations are not a git concept.
No; just knowing how git works is enough to understand that force-pushing squashed commits or removing branches on remote will not necessarily remove the actual data on remote.
GitHub API (or just using the web UI) only makes these features more obvious. For example, you can find and check commit referenced in MR comments even if it was force-pushed away.
> was deeply surprised about GitHub's brute-forceable short commit IDs
Short commit IDs are not GitHub feature, they are git feature.
> If I use GitHub's web interface to click myself a repository with complex access rights, I assume they have concepts in place to thoroughly enforce these access rights.
Have you ever tried to make private GitHub repository public? There is a clear warning that code, logs and activity history will become public. Maybe they should include additional clause about forks there.
And it's completely reasonable to assume no one external to the org when it was private would have those hashes.
It sounds like github's antipattern here is retaining a log of all events which may leak these hashes, and is really not an assumption I'd expect a git user to make.
They're a local feature sure. But you already have a list of local commits, just open the .git directory.
Can you connect to a vanilla git server and enumerate every single hash?
> Maybe they should include additional clause about forks there.
It would help but they need much more than a clause about forks.
Ideally they would purge that extra data when making something public.
If you have ssh access yes, but I don't think you can do this with just git (and of course github doesn't provide ssh access to the git repo servers)
The public distribution of commit hashes via their event log seems really irresponsible on github's part to me.
"Why not just...". Once you already know something it can seem obvious.
There are some user-facing parts: You can find the fork network and some related bits under repo insights. (The UX is not great.)
https://github.com/apache/airflow/forks?include=active&page=...
My conclusion would be: don’t use GitHub.
I 'love' it when I see the words "hide", "archive", "remove", and other newspeak to avoid using the word "delete", since 'they' never actually delete (plus there are 1-2-5-10-forever years' of backups where your 'deleted' info can be retrieved relatively easy).
> After some internal discussion, we have determined this is a known low risk issue. We may make this functionality more strict in the future, but don't have anything to announce now. As a result, this is not eligible for reward under the Bug Bounty program. Below is a reference to our instructions for users to remove sensitive data from a repository. https://help.github.com/articles/removing-sensitive-data-fro...
Here is their full response from back then:
> Thanks for the submission! We have reviewed your report and validated your findings. After internally assessing the finding we have determined it is a known low risk issue. We may make this functionality more strict in the future, but don't have anything to announce now. As a result, this is not eligible for reward under the Bug Bounty program.
> GitHub stores the parent repository along with forks in a "repository network". It is a known behavior that objects from one network member are readable via other network members. Blobs and commits are stored together, while refs are stored separately for each fork. This shared storage model is what allows for pull requests between members of the same network. When a repository's visibility changes (Eg. public->private) we remove it from the network to prevent private commits/blobs from being readable via another network member.