I don't think this is a support option.
You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions.
> In fact, this is not a supported option in openssh.
This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.
AIUI OpenSSH does not provide any way to use PKCS11 to protect host keys, which are the concern here.
You can use PKCS11 to sign OpenSSH certificates, so if GitHub had elected to use certificates here, it could have protected the CA keys for those certificates in an HSM, but it did not.
How can openssh documentation be vendor-specific?
Or are you saying that vendors commonly provide an openssh fork/patchset/plugin allowing for HSM-resident host keys?
I isn't, because the cryptography is (in case of HSM) not handled by OpenSSH itself. So OpenSSH's configuration has nothing to do with the HSM.
Usually, the actual cryptographic functions are not performed user-space, but handled by the kernel, which in turn can offload this to dedicated hardware. Basically if you compile OpenSSH for it to use kernel level TLS, then OpenSSH can work with a HSM without it even knowing it.
Disclaimer: this is simplified explanation, there is a lot more to this, and I am by no means an expert on this matter.
I‘m aware of the PKCS#11 integration in the OpenSSH client and have dabbled a bit with it but was not aware of any server side equivalent.
And how does TLS fit in here? SSH is a very different protocol from that, no?
Update: I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support or kernel-mode cryptography (which also in the case of Linux only addresses symmetric encryption to my knowledge, at least the mainline kernel version I know of).
Maybe you're thinking of some other SSH implementation? The protocol definitely allows for server-side HSM usage, and Github at their scale is not bound to OpenSSH in any way.
It makes a lot of sense, since it avoids having to link the HSM/PCKS#11 stuff against sshd.
OpenSSHd talks to an ssh-agent that then talks to the HSM:
> Identifies the UNIX-domain socket used to communicate with an agent that has access to the private host keys. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be read from the SSH_AUTH_SOCK environment variable.
There is the HostKeyAgent configuration directive, which communicates over a unix domain socket to make signing requests.
https://framkant.org/2017/10/strong-authentication-openssh-h...
https://github.com/openssh/openssh-portable/blob/12492c0abf1...
Agreed. I have seen some crazy stuff in the payment card industry. I can't recall what I can and can't talk about so I'll just say "Atalla".
https://framkant.org/2017/10/strong-authentication-openssh-h...
It looks like they currently use the `ssh_bind_options_set` function with `SSH_BIND_OPTIONS_HOSTKEY` to set the host keys which means they exist on disk at some point. HSM aside, I believe it would be possible to use the `ssh_bind_set_key` and deserialize them from a secret vault so they only exist in the memory of the ssh daemon.
Obviously they also just straight up have enough resources to fork the code and modify it to use an HSM.
Source: looking at their ssh server portion of `babeld` in ghidra right now as part of hunting for bug bounties.
For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?
Not just nation state actors, but basically anyone in a position to MITM.
Also, you don't have to be a nation state actor to extort a GH employee. Any bad guy can do a "give me this key or I'll hurt your kid". People are being extorted for a lot less.
There are billions of dollars of assets flowing through GH's infrastructure, for the sake of safety (!= security) of Github's employees, nobody should ever have access to key material.
Do you mean source code here? I have a hard time believing source code holds that much value.
If there are ~350 million private repos then they'd only need to be worth an average of $30 each to be worth a billion dollars in total. Which doesn't seem farfetched.
For proof, try searching for a mundane string in GH Code search. The vast majority of repos you see will be basically garbage.
The host key is the only thing ensuring you’re actually talking to GitHub.com when you push code.
To add to sibling comments, it should not have been possible to make this mistake. That it was possible is concerning.
Great! Then I can communicate confidentially with whomever is MITM'ing me.
/s
Do you realize that the code just sits on GitHub servers even if it's private?
If you have any degree of paranoia, why do you put your code into GitHub?!?!
Like, if you work on code which
ISO/IEC 27001:2013 doesn’t say you have to store private keys in HSMs? It just requires you to have a standards compliant ISMS that implements all Annex A controls and all of the clauses. Annex A and the clauses don’t specifically mandate this.
If you can convince an auditor that you have controls in-place that meet the standard for protecting cryptographic media you basically meet the standard. The controls can be a wide variety of options and don’t specifically mandate technical implementation details for many, many things.
You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Just because your auditor would red flag you doesn’t mean all auditors would. The standard is only as effective as the weakest, cheapest auditor, and there are perverse incentives that make auditors financially incentivized to certify companies due to recurring revenue.
Thanks for the insight, good advice.
But also from the same GH article [0]:
The ISO 27001 certification is the latest addition to GitHub’s compliance portfolio, preceded by SOC and ISAE reports, FedRAMP Tailored LiSaaS ATO, and the Cloud Security Alliance CAIQ.
Do you have any knowledge on one of these certifications (for exmaple FedRAMP) that puts any restrictions on handling key material?
[0] https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
It took us ~2 years of back & forth with various parties & auditors to get per-client exclusions for accepting end-customer debit PIN codes in-branch on an iPad screen. These banks do not have fully-compliant solutions and must have exceptions on file.
As the adjacent commentor 1970-01-01 states, PCI DSS is actually pretty strict and requires use of HSMs. However, that level of PCI compliance is only required for institutions that actually handle the full credit card number. GitHub uses Stripe as a payment gateway, so they don't need to meet it.
FIPS 140-3 levels 3 and 4 often are met by using HSMs, but technically speaking there aren't any standards that I know of that exist outside of the payments industry that have hard requirements for HSM use for all cryptographic key media.
I think the unfortunate reality is that many organizations struggle to deploy HSMs widely for all cryptographic media because they aren't very scalable and deployment can lead to other operational constraints that many companies can't or won't deal with. It's much easier for low-frequency high-importance tasks like signing new releases of OS images or packages, rather than I/O heavy high-frequency operations for a site like GitHub.
So, in-summary, look for PCI DSS or FIPS 140-3 Levels 3 and 4, but, be prepared to discover that "creative" solutions may let a company meet even the highest levels of FIPS compliance without HSMs for all cases.
I know it's sucky advice, but for ISO in-particular, I suggest just acquiring and reading 27001 if you want to use it as a basis for decision making. It does offer a lot, and I think it's a very well-written standard, but all of the implementation advice is in ISO 27002, and it's not required. The advice in 27002, when used to meet 27001, leads to a very compelling program. But a 27001-compliant org can completely forego 27002 and DIY it as long as it meets the test criteria set-out by the auditor.
Edit: Also, every auditor will include standard disclaimers in their report that they perform sampling-based testing, and that the testing is not a complete guarantee of the state of the company. ISO in-particular is performed by getting an initial audit, followed by two years of surveillance audits that test the entire suite of controls. But due to sampling-based methodology, something can be overlooked or evidence can be provided that doesn't holistically reflect the org in all cases. If it's any consolation, this particular issue will certainly be in their audit for next year, as a security incident and probably lead to an opportunity-for-improvement from the auditor.
So you may be able to get certified for policy, process but not necessarily the programming to allow something in the first place.
Hopefully now knowing this will improve under Microsoft.
For those who value this, self- hosting or on-prem instances of git might shoot up in importance.
ISO 27001 certification does not require you to put keys into an HSM. The standard requires you to have controls in place, be aware of your risks and to maintain a risk register. But in no way does the standard require HSM's.
The standard would even be OK with storing this on a floppy drive if the risks surrounding that were identified and mitigated (or accepted).
Not to diminish the problems of having a large entity like Github handle a private key like that, but if that was your level of paranoia, you probably should have used commit signatures all along and not relied on Github to do that job for you.
I also don't want to diminish the concerns around Github or similar orgs losing control of a private key, but the far more realistic concern for the vast majority of threat models is often put to the wayside in favor of what amounts to a scary story. Rather than the straightforward key removal and replacement that this should be, I (and surely many others) have spent all morning combatting this specific FUD that cropped up on HN with leadership and many engineers. It's actually quite detrimental to quickly remediating the actual concerns introduced by this leak.
I understand that security inspires people to be as pedantic as possible - that's where some big exploits come from on occasion - but I really hope the average HN narrative changes toward "what is your actual, real-world threat model" vs. "here is a highly theoretical edge-case scenario, applicable to very few, that I'll state as a general fact so everyone will now wonder if they should spend months auditing their codebase and secrets". Put simply: this is why people just start ignoring security measures in the real world. Surely someone has already coined the term "security fatigue".
It's all just a bit unbalanced, and definitely becomes frustrating when those suggesting these "world is burning" scenarios didn't even take the available precautions that apparently would satisfy their threat model (i.e. commit sigs, as you suggested)
Ok, end rant :)
Having a correct threat model is the first step towards building reasonable security controls. But far too many are willing to pander to the "It rather involved being on the other side of this airtight hatchway" [0] scenarios.
[0] https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
1) Nation state level actors can probably insert or compromise high level staff, or multiple high level staff, at any given company, and perform MITM attacks fairly easily. And some could compel turning over your code or secrets more directly anyway. Not worth worrying about this scenario: nobody working on anything truly sensitive should be using any externally hosted (or even externally networked) repositories.
2) It is much more difficult for other actors to do a MITM attack, and if they did, they'd probably have access to your code more directly.
3) Your code actually isn't worth much to anybody else. Imagine someone launching a complete clone of HN or any other site. Who cares? Nobody. What makes your code valuable is that you have it, and that you have a network and relationship with your customers. If somebody stole my company's codebases, I'd feel sorry for them, that they are going to waste any time wading through these useless piles of shit. The only potential problems are if secrets or major vulnerabilities are exposed and provide a path for exploit (like ability to access servers, services, exposing potential ransomware attacks).
This leak opened a time window big enough for that to happen. We may or may not know if it did. I doubt this info would be offered to the public because it would sink the business.
The cloud is always the convenience of someone else’s computer over some amount of security.
Requiring all commits to be signed by trusted keys avoids the risks associated with someone tampering with a repository hosted on GitHub if they are able to get access to it, although it doesn't protect against code being leaked.
See here for details: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
The risk of disclosure is pretty obvious with GitHub, and I’d assume anyone with low risk tolerance here is using something else, including the on-prem GitHub. I can think of a dozen higher risks.
Many signed commits ("Verified" in green) on GitHub are signed with GitHub's own private key¹, not the committer's key. There's a good technical reason, but many committers don't realise their own signing key isn't the one used on their signed commits to the main branch.
That GitHub private key would be a fun one to have leaked!
It would invalidate most of the "Verified" flags on commits on most repo main branch histories.
(¹ GitHub uses GitHub's own commit-signing key when you use the GitHub GUI to merge or squash-merge commits if you already signed them, and the resulting commits show as "Verified" the same as conmits signed by your own key. So you do have to sign with your own key,, but it's not the key used on the main branch commits everyone sees whrn they visit and doenload. Many controlled workflows require approved PRs to be merged to main/master through GitHub's tools, and many users default to using the GUI for PR merges by choice.)
Number of people just not blindly using TOFU with github over ssh must be quite low.
Who here went here https://docs.github.com/en/authentication/keeping-your-accou... and added the keys manually to known_hosts before using github for the first time?
Oh, I missed the "not" the first time I read it, which changed your overall meaning entirely. My bad.
I think this suggests we need more information from github. For instance GH employees may not always have had live access to this key, this could have happened as part of an operation that gave temporary access to an employee only recently. Or it could have been stored plaintext on multiple employees' home computers since creation.
When was the leaked key created anyway?
I suppose (Open?)SSH's PKI mode could support a model like that, but as others have noted here, this requires much more manual work on the user's side than comparing a TOFU key hash.
Maybe that model could be extended to allow TOFU for CAs, though? But I think PKI/CA mode is an OpenSSH extension to the SSH protocol as it is, and that would be a further extension to that extension...
Storing the key in some kind of credential vault that can only be accessed from the hosts that need it at startup would usually be enough to prevent this particular kind of error (unless you're giving root on those boxes to people without enough sense to avoid checking private keys into git, in which case you've probably got worse problems).
Regarding the secrecy of the data you host at github, you should operate under the assumption that a sizeable number of github employees will have access to it. You should assume that it's all sitting unencrypted on several different disk platters replicated at several different geographically separated locations. Because it is.
One of the things that you give up when you host your private data on the cloud is controlling who and under what circumstances people can view or modify your private data. If the content of the data is enough to sink you/your company without remedy you should not store it on the cloud.
[1] Which is sort of tautological and silly, because that's true of all sites and all host keys. What the comment was trying to imply was that the choice of storage of this key invalidates any trust we might have in GitHub/Microsoft regarding key management, and that therefore we shouldn't trust them. Which is also tautological and silly. Either you trust them or you don't, that's not a technological argument.
When we signed on with GH as a paying customer over a decade ago, they were quite clear that private repos should not be considered secure storage for secrets. It’s not encrypted at rest, and GitHub staff have access to it. It takes only a few clicks to go from private to public.
The real question is how many services are able to access the data live and how many support and debug interfaces (indirectly) allow you to read it. Measure GitHub's success in securing the secrecy of private repos in how few employees can breach it without causing alarms. Even without cynicism I would be surprised if it was their main concern. Data integrity is far more important for code. (There are notable exceptions, of course. If applicable, don't put it in the cloud!)
There are plenty of tools to either keep them encrypted (we just use simple GPG, but our team is small) or just auto-generate and never show to user in the first place (various key vaults that can be used from automation like HashiCorp's Vault)
If ssh token based auth is used, it's different of course, because then the server gets access to the token. Ideally Github would invalidate all their auth tokens as well.
The fun fact is that a token compromise (or any other attack) can still happen any point in the future with devices that still have outdated ssh keys. That's a bit unfortunate as no revocation mechanism exists for ssh keys... ideally clients would blacklist the ssh key, given the huge importance of github.
E.g. keybase could do X.509 like Certificate Transparency.
:
$ keybase pgp -h
NAME:
keybase pgp - Manage keybase PGP keys
USAGE:
keybase pgp <command> [arguments...]
COMMANDS:
gen Generate a new PGP key and write to local secret keychain
pull Download the latest PGP keys for people you follow.
update Update your public PGP keys on keybase with those exported from the local GPG keyring
select Select a key from GnuPG as your own and register the public half with Keybase
sign PGP sign a document.
encrypt PGP encrypt messages or files for keybase users
decrypt PGP decrypt messages or files for keybase users
verify PGP verify message or file signatures for keybase users
export Export a PGP key from keybase
import Import a PGP key into keybase
drop Drop Keybase's use of a PGP key
list List the active PGP keys in your account.
purge Purge all PGP keys from Keybase keyring
push-private Export PGP keys from GnuPG keychain, and write them to KBFS.
pull-private Export PGP from KBFS and write them to the GnuPG keychain
help, h Shows a list of commands or help for one command
$ keybase pgp drop -h
NAME:
keybase pgp drop - Drop Keybase's use of a PGP key
USAGE:
keybase pgp drop <key-id>
DESCRIPTION:
"keybase pgp drop" signs a statement saying the given PGP
key should no longer be associated with this account. It will **not** sign a PGP-style
revocation cert for this key; you'll have to do that on your own.
- "Revoked a PGP key, is there any way to get a revocation certificate now?" https://github.com/keybase/keybase-issues/issues/2963
- "Overview of Certification Systems: X.509, CA, PGP and SKIP"
...
- k8s docker vault secrets [owasp, inurl:awesome] https://www.google.com/search?q=k8s+docker+vault+secrets+owa... https://github.com/gites/awesome-vault-tools
- Why secrets shouldn't be passed in $ENVIRONMENT variables; though e.g. the "12 Factor App" pattern advises to parametrize applications mostly with environment variables that show in /proc/pid/environ but not /proc/pid/cmdline
W3C DID supports GPG proofs and revocation IIRC:
"9.6 Key and Signature Expiration" https://www.w3.org/TR/did-core/#key-and-signature-expiration
"9.8 Verification Method Revocation" https://www.w3.org/TR/did-core/#verification-method-revocati...
Blockerts is built upon W3C DID and W3C Verified Credentials, W3C Linked Data Signatures, and Merkel trees (and JSON-LD). From the Blockerts FAQ https://www.blockcerts.org/guide/faq.html :
> How are certificates revoked?
> Even though certificates can be issued to a cohort of people, the issuer can still revoke from a single recipient. The Blockcerts standard supports a range of revocation techniques. Currently, the primary factor influencing the choice of revocation technique is the particular schema used.
> The Open Badges specification allows a HTTP URI revocation list. Each id field in the revokedAssertions array should match the assertion.id field in the certificate to revoke.
Re: CT and W3C VC Verifiable Credentials (and DNS record types for cert/pubkey hashes that must also be revoked; DoH/DoT + DNSSEC; EDNS): https://news.ycombinator.com/item?id=32753994 https://westurner.github.io/hnlog/#comment-32753994
"Verifiable Credential Data Integrity 1.0: Securing the Integrity of Verifiable Credential Data" (Working Draft March 2023) > Security Considerations https://www.w3.org/TR/vc-data-integrity/#security-considerat...
If a system does not have key revocation it cannot be sufficiently secured.
Well, at least not without encryption that is under your control.
By no means do I want to see the dev get fined or blackballed from the industry.
But if there is any 1 person responsible, it’s the person who did it. The reason why the dev shouldn’t be fined/blackballed is because it’s not just 1 person’s fault. I mean, fining or booting his manager out of the industry? Really?
Github's customers trust GH/MS with their code, which, for most businesses, is a high value asset. It wouldn't surprise me if this all results in a massive lawsuit. Not just against GH as a company, but also to those involved (like the CISO). Also, how on earth was it never discovered during an audit that the SSH private key was plain text and accessible? How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM?
Obviously, as a paying customer I am quite angry with GH right now. So I might be overreacting when I write this, but IMO the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license.
[0] https://github.blog/2022-05-16-github-achieves-iso-iec-27001...