Why would that be unknown? GitHub has HTTP and SSH access logs, right?
Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key?
The answer is 31337.
To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
- if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable.
- rotating the key is the only correct reaction to that
- they should have better communications in place to help users mitigate MITM
- there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly
- on principle this speculation makes sense, but understanding the threat makes it moot
- you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
Thus rotating is the only logical course of action.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.
You just ignore it and hope for the best ?
Only if you are certain (and better be really sure you haven't missed any cache/cdn, temp files backus etc.) it wasn't accessed you do nothing.
You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.
I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background.
Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs.
Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
Blockchain is a perfect solution to this. I wonder why it is not considered yet.
Also it's way better than TOFU, you can just add the CA key to known_hosts and avoid TOFU for each machine.
(Nevermind that you'll probably not accidentally commit some semi-ephemeral host key that's rotated often somewhere, because it will not be some special snowflake key you care about, but something handled by your infrastructure software automatically for each machine)
How easy is it to rotate the keys of your CA?
I hope you don't need that SSH connection to fix your broken CRL endpoint!
We need to compare the relative risks within the same context, namely within a company like GitHub
So it's not relevant to bring up failures of other CAs
Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
We also have a way to put SSH host key fingerprints in DNS records already.
I'd guess that most systems aren't using DoH/DoT or end-to-end DNSSEC yet. Some browsers do, but that doesn't help tooling frequently used on the command line.
There could be a well known URL that enables people to fetch ssh keys automatically in a safe manner.
So that any large entity can own your servers with easy. (Well, they already can, but not through this vulnerability.)
Anyway, the only thing CAs do is to move that prompt into another, earlier time. It's the same prompt, the same possibility for MITM, and the same amount of shared trust to get wrong. You just add a 3rd party that you have to trust.
SSH does have a CA system. Anybody that isn't managing a large datacenter will avoid it, for good reason.
Eh, let's not pretend existing SSL certificate validation is anything to write home about.
Even without any ephemeral servers involved, barely anybody is validating cert fingerprints on first use.
And among people using ephemeral servers, 99% of applications have either baked a certificate into their image (so that any compromised host means a compromise of the critical, impossible-to-revoke-or-rotate key) - or every new server gets a new cert and users have either been trained to ignore certificate warnings, or they've disabled strict host key checking and their known hosts file.
The existing SSL cert validation options are perfect if you're a home gamer or you're running a few dozen bare metal servers with all your SSL users within yelling distance in the same office. But we all know it's a joke beyond that.
There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
edit: I found this helpful and honestly had no idea I should be doing this (I’m a hobbiest not a professional) https://bitlaunch.io/blog/how-to-check-your-ssh-key-fingerpr...
Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM) because Github, their college or random person on internet said that that's what you have to do to get rid of error.
This is less likely because unlike for TOFU the SSH client just rejects the mismatch and insists you take manual action, and the likely manual action will be "Paste this stuff from the announcement".
So an adversary needs to either subvert whatever messaging you see (which is tricky, likely impossible for a random user visiting the github web site wondering what's wrong) or hope that you just try to muddle along and do TOFU again, putting you in the same spot as a whole bunch of users every day at GitHub.
The problem here is that their first command that they advise using is the one that removes the old key and most users are just going to stop right there because it solves the problem of getting the key warning.
The right solution here is to provide a command that most users are going to copy and paste that deletes the old key and adds the new key all at once.
If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame. If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't you verify the key out of band?
The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
One way to solve this in TOFU is to have a time window where both keys are presented.
well, yes
github doesn't accept https push any more
Maybe you’re referring to how they no longer accept passwords for HTTPS auth? You have to auth for HTTPS push with a personal access token.
I get a bit paranoid when having to deal with Tokens on various CI/CD environments as it stands. And the things that start breaking every year when I forget to update them. Note: this is personal/hobby projects, not corporate stuff, where I'm strictly in the codebase and try to keep my fingers out of CI/CD beyond getting a Docker image built, and someone else configures the keys/auth.
I sure did. Doesn’t everyone?
So MITM for some of 50m users is strictly better than MITM for all of 50m users.
How do you know this?
Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
> leaked in public repo
Me: Yeah, that's why they are doing it.
"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."
Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.