News for hackers. Stuff that matters.
Re:New Intermediate Certificates by phantom784 on Tuesday March 19, 2024 @11:21AM (#39756684)
Interesting that they wish to discourage key pinning. This is pretty commonly used by apps to make them harder to reverse engineer. It prevents you from installing your own cert in the system's trust store and then using that to man-in-the-middle the app's communication with its backend.
Re:New Intermediate Certificates by mcpherrinm on Tuesday March 19, 2024 @11:28AM (#39756778)
It's important to note that this is preventing pinning /intermediates/ which are re-issued about annually. It is usually a mistake to do that. It is possible that due to a large-scale incident, we'd have to revoke an intermediate immediately and switch to a backup.

While we don't recommend key pinning, there's nothing to prevent pinning Let's Encrypt's root CAs.

(I work for Let's Encrypt)

Re:New Intermediate Certificates by foobiekr on Tuesday March 19, 2024 @11:38AM (#39756883)
exactly right.
Re:New Intermediate Certificates by AlwaysNewb23 on Tuesday March 19, 2024 @12:52PM (#39757723)
Thanks for clarifying this.
Re:New Intermediate Certificates by chaz6 on Tuesday March 19, 2024 @03:08PM (#39759160)
Do clients generally check for revocation for every cert in the chain? You would think you would definitely want to as a compromised root cert would be highly damaging!
Re:New Intermediate Certificates by mcpherrinm on Tuesday March 19, 2024 @05:18PM (#39760316)
There's unfortunately a lot of nuance and sadness here.

Generally, a root is trusted by virtue of being in your root store and can't be revoked via OCSP/CRLs. An operating system security update should remove distrusted roots.

The mainstream browsers all have a push-based mechanism that they can use to rapidly revoke a root or intermediate faster than that. If something truly bad happened, Let's Encrypt would ensure that happened ASAP.

However, many clients do no revocation checks at all. That's a big part of why the ecosystem is pushing to shorter lived certificates, intermediates, and roots.

Re:New Intermediate Certificates by tialaramex on Tuesday March 19, 2024 @05:35PM (#39760495)
It doesn't really mean anything to have "revocation" of a Root. The way forward is to explicitly cease trusting this Root if you believe it has been compromised. For example Microsoft can ship a Windows Update which distrusts roots they by default trust in Windows.

Exactly what clients do about revocation for Intermediates and End Entity certificates varies, the revocation information will be available publicly in one (or both) of two forms:

The Online Certificate Status Protocol. OCSP results have a lifetime so you can check back periodically, and they're created via a distinct key from the Root, which has its own certificate (provided in the response) showing that it's trusted to do this specific work.

Certificate Revocation Lists are signed documents with a list of revoked certificates. CRLs are again signed using a distinct key with its own certificate showing that it was authorised to sign these lists.

Some client vendors may (semi)automatically aggregate information to produce their own summaries so that their clients can rely on the aggregate rather than fetching revocation information during use.

OCSP in particular was designed so that it could be stapled which is a technique where your server fetches and then includes the OCSP results for your end entity cert (and possibly the intermediate) with the cert itself during a connection handshake, the client thus now has a valid (albeit perhaps not entirely fresh) OCSP answer without doing its own checks, since OCSP is signed you can't forge this although of course if you have a 48 hour OCSP good answer you could continue playing it for up to 48 hours even if subsequent OCSP results said revoked.

Unfortunately implementation for OCSP stapling has been fairly poor, it's usually wrongly or at least badly implemented by people who apparently have no idea what the goal was and so it can make your service needlessly less reliable if your server is defective in this way, so the main browsers are not (last I knew) much interested in pursuing this further.

Re:New Intermediate Certificates by phasmantistes on Tuesday March 19, 2024 @11:31AM (#39756806)
It's specifically to discourage intermediate key pinning. If folks want to pin their own end-entity public key (and always re-use the same key when renewing their cert), go for it -- dealing with compromise of their own key is their own problem to solve. Or if they want to pin a root public key to ensure some other CA doesn't issue a MITM certificate, go for it (although that doesn't prevent a bad actor from getting the same CA to issue a MITM certificate; there are other mechanisms to prevent that).

Just please don't pin intermediate CA keys, which should be opaque to the end-user and need to be able to change quickly without breaking a bunch of apps.

Re:New Intermediate Certificates by agwa on Tuesday March 19, 2024 @11:42AM (#39756935)
> Or if they want to pin a root public key to ensure some other CA doesn't issue a MITM certificate, go for it

Please don't pin roots, as that makes it harder to distrust CAs, reducing the agility of the WebPKI. See the Symantec distrust for a painful example.

Chrome and Firefox will be introducing term limits on roots in the near future, which will hopefully help to discourage this harmful practice.

Re:New Intermediate Certificates by gray_-_wolf on Tuesday March 19, 2024 @12:33PM (#39757515)
> Please don't pin roots

So what would be the recommended way to protect against government MitM by using some obscure CA?

Re:New Intermediate Certificates by schoen on Tuesday March 19, 2024 @12:40PM (#39757576)
Currently CT logs:

https://certificate.transparency.dev/

Monitor CT for your domain name and if you find an "obscure CA" misissuing for your domain, report it! This may result in the obscure CA getting explicitly distrusted.

At some level this is not that great a solution, but it's really so much better than what we had 13-14 years ago.

Re:New Intermediate Certificates by thrtythreeforty on Tuesday March 19, 2024 @03:15PM (#39759244)
I'm curious: What's the defense mechanism for a dodgy CA not publishing a CT log entry for the misused domain?
Re:New Intermediate Certificates by woodruffw on Tuesday March 19, 2024 @03:29PM (#39759379)
A CT-honoring client should reject an end-entity certificate that isn't accompanied by a SCT. In other words: a dodgy CA that skips CT to avoid disclosure of their mis-issued certificate should be unable to convince any CT-honoring client to accept that certificate.
Re:New Intermediate Certificates by notachatbot123 on Tuesday March 19, 2024 @12:52PM (#39757725)
An adversary might MITM the CT as well.
Re:New Intermediate Certificates by tptacek on Tuesday March 19, 2024 @01:14PM (#39757996)
How?
Re:New Intermediate Certificates by cpach on Wednesday March 20, 2024 @11:06AM (#39767666)
:crickets:
Re:New Intermediate Certificates by michaelt on Tuesday March 19, 2024 @04:48PM (#39760048)
CT logs only mean that if after the horse has bolted, you can start the process to close the door within hours.

That's adequate for low-value targets, but it's hardly sufficient for high-value targets. If a CA mis-issued a cert for *.aws.amazon.com should we be waiting around for a manual process on some mailing list to invalidate it?

Re:New Intermediate Certificates by schoen on Wednesday March 20, 2024 @02:16PM (#39770163)
I agree conceptually, and I think it's sad that we don't have anything more absolute. (I was a fan of HPKP, which has been deprecated as a "footgun" because apparently users often didn't understand what it was doing or weren't cautious enough when using it.)

But the CT system seems to work very well in practice. While the detection and remedy part is awkwardly manual, there are people working hard on them. There are also (following a "you have to disclose all intermediates ahead of time" rule introduced by Mozilla) fewer intermediates and we actually have a list of them.

https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRe...

Detected incidents involving intentional misissuance are very rare. When unintentional misissuance happens, the responsible CA has to publicly explain how it happened and what it will do to prevent the situation from happening again.

Re:New Intermediate Certificates by xg15 on Tuesday March 19, 2024 @06:32PM (#39761002)
And then what? What are you supposed to do in the meantime until browsers have decided to distrust the CA?

Also, how do you detect "obscure" CAs in the log? If your CT monitoring daemon has some logic to distinguish "good" from "bad" CAs, why not use that logic directly in the app and implement certificate pinning?

Re:New Intermediate Certificates by josephcsible on Tuesday March 19, 2024 @12:32PM (#39757497)
You should be able to fully inspect all traffic originating from your device and reverse-engineer everything running on it, so any change that makes it harder for others to stop you from doing so is a good one.
Re:New Intermediate Certificates by jcalvinowens on Tuesday March 19, 2024 @11:42AM (#39756928)
> Interesting that they wish to discourage key pinning. This is pretty commonly used by apps to make them harder to reverse engineer

I don't think you'd typically pin intermediates if that was your goal: you'd pin the site cert you control. It could be as simple as a whitelist of certificate SHAs.

Re:New Intermediate Certificates by tialaramex on Tuesday March 19, 2024 @12:20PM (#39757325)
I'd recommend pinning (your) keys because knowing cert SHAs ahead of time means you need the certificates themselves and in a nasty situation the CAs you relied on to issue those certs may be out of the picture, if you're pinning a key that's fine, CA #5 will cheerfully issue you a brand new certificate against the same key you were using with CA #1 before something went badly wrong - but with cert SHAs you have to bootstrap all of that fresh if it happens.

If you pin keys you can even pin a key you haven't and never plan to use, keeping the corresponding private key in the company safe as a hedge against something going badly wrong.

Re:New Intermediate Certificates by jcalvinowens on Tuesday March 19, 2024 @12:33PM (#39757508)
I completely disagree with you: reusing private keys is an enormous vulnerability. It's really important to rotate them.

It's very easy to avoid the pitfall you mentioned by having multiple valid certs with different expiry dates. You can easily use multiple CAs.

Done your way, a single leaked private key means your entire site is compromised indefinitely. That's unacceptable to me.

Re:New Intermediate Certificates by tomputer on Tuesday March 19, 2024 @04:31PM (#39759921)
> reusing private keys is an enormous vulnerability.

As long as the private key is stored/handled safely and RSA/ECC is not broken, it is not vulnerable.

I do agree that key rotation is better/recommended practice.

> a single leaked private key means your entire site is compromised

The leak is the actual vulnerability. As long as the leak is still there and you are not aware of the compromised private key, a fresh new private key will probably leak again.

However, the chances of leaking may be greater if a private key has to be used in multiple locations.

Re:New Intermediate Certificates by toast0 on Tuesday March 19, 2024 @12:43PM (#39757607)
Pinning the site cert (or its key) is not great because it makes it very hard to rotate your certificate or key.

And you've got to be able to do that, in case your key is disclosed or possibly disclosed.

Pinning intermediates doesn't make a lot of sense; unless you have a deep relationship with the CA, they can and do change intermediates, and may not be able to issue from the old intermediate after the change. But, I've been involved in cert pinning where we had to pin intermediates, because that's the information the client API returned.

Pinning roots makes sense, but you need to pin more than one. And roots from more than one organization. And you need a succession plan. You can't use a new root on an existing hostname until all of the deployed clients with pins that could use that hostname have become irrelevant. If your all your trusted CAs stop issuing, you're pretty screwed.

You can sometimes use an in-house root, which makes continuity easier, but not if you need to share hostnames with browsers. You can't get an entity cert signed by two issuers, so there's no way to present a cert that is trusted by browsers and also signed by your root. You can present alternate paths for the issuers of your cert (and any intermediates) and if the x.509 libraries do the right thing, any valid path is accepted; but not all libraries do the right thing, and again, that doesn't apply to the entity cert.

Re:New Intermediate Certificates by jcalvinowens on Tuesday March 19, 2024 @12:46PM (#39757655)
Absolutely, I agree with everything you said.

Pinning is not an effective anti-reverse-engineer technique. It's just annoying obfuscation to anybody who knows what they're doing.

Re:New Intermediate Certificates by toast0 on Tuesday March 19, 2024 @01:23PM (#39758125)
Pinning isn't really for anti-revere-engineering. It does discourage some reverse engineers, which is a bonus. But it's main purpose is to protect data integrity between the client and the server --- there's plenty of cases of CAs in system root stores that issued certs that could be used to intercept and manipulate traffic. There's also plenty of cases where something installed on the user's device adds itself to the system root store and intercepts traffic with the intent of passing it through but does not do so correctly and causes hard to diagnose errors.

A pinned cert makes it easy to detect these MITMs and refuse to work if they're in place. A dedicated reverse engineer is "just" going to patch out the pin, and go on with their day, no big deal. But ISP MITM or corporate MITM or on device MITM is going to be stopped, and that's worthwhile.

Re:New Intermediate Certificates by jcalvinowens on Tuesday March 19, 2024 @02:13PM (#39758654)
That's sort of outside the threat model though, isn't it? If an attacker can install a root cert, the entire thing is compromised.

If you're concerned about that and want to pin, why have a chain of trust at all? Couldn't you just embed self-signed certs in the binary at that point?

If you mean preventing users from clicking through the SSL warnings in a browser if they're being MITM'd, that makes sense to me.

Re:New Intermediate Certificates by toast0 on Tuesday March 19, 2024 @03:14PM (#39759227)
> If you're concerned about that and want to pin, why have a chain of trust at all? Couldn't you just embed self-signed certs in the binary at that point?

Not if you need to serve https traffic to clients and browsers from the same servers / same hostnames.

> If you mean preventing users from clicking through the SSL warnings in a browser if they're being MITM'd, that makes sense to me.

That comes from HSTS, not from cert pinning. Cert pinning in browsers is only available if you can convince the browser to include it in their static lists, which means you've got to be big... so that is what it is.

Re:New Intermediate Certificates by jcalvinowens on Tuesday March 19, 2024 @04:28PM (#39759908)
> Not if you need to serve https traffic to clients and browsers from the same servers / same hostnames

That's what I was getting at, that makes sense to me.

> That comes from HSTS, not from cert pinning.

HSTS is a terrible half assed solution that relies on adding domains to magical predistributed lists. I wouldn't trust it if I cared enough about MITM to go to all this trouble.

Re:New Intermediate Certificates by toast0 on Tuesday March 19, 2024 @05:02PM (#39760178)
> HSTS is a terrible half assed solution that relies on adding domains to magical predistributed lists. I wouldn't trust it if I cared enough about MITM to go to all this trouble.

Sure, but cert pinning in browsers is done with the same magical redistributed lists; it's just even harder to get them changed.

Re:New Intermediate Certificates by michaelt on Tuesday March 19, 2024 @11:30AM (#39756801)
Does this not simply mean they want people to pin "ISRG Root X1" instead of pinning "Let’s Encrypt R10" because they want the ability to rotate the latter?

Although IMHO saying "don't pin this cert which expires in 2027, instead pin this cert which expires in 2035" is kinda a weird distinction to make as those are both too short.

Re:New Intermediate Certificates by londons_explore on Tuesday March 19, 2024 @12:12PM (#39757231)
I kinda wish pins by default became unpinned when the pinned cert expired.

Pinning an expired cert means you are 100% sure the software won't work.

Re:New Intermediate Certificates by Wowfunhappy on Tuesday March 19, 2024 @12:26PM (#39757430)
What prevents the user from simply setting their clock forward?

(I do agree cert pinning is bad overall.)

Re:New Intermediate Certificates by patmorgan23 on Tuesday March 19, 2024 @12:52PM (#39757718)
If the user wants to ignore cert error they're going to ignore cert errors
Re:New Intermediate Certificates by michaelt on Tuesday March 19, 2024 @01:55PM (#39758484)
Pinning your CA's certificate protects against two things:

1. All-powerful NSA types who've managed to steal a CA's private key or get a certificate mis-issued. It's a nigh-irreplaceable attack opportunity, but the moment they use it it'll be revoked within hours, so naturally it's reserved for the absolute highest value target. For example, [1]

2. Users trying to reverse-engineer your app.

You can imagine which of these is the most common - perhaps you don't want the user to be able to bypass the pin, even intentionally.

[1] https://www.eff.org/deeplinks/2011/08/iranian-man-middle-att...

Re:New Intermediate Certificates by ijustlovemath on Tuesday March 19, 2024 @11:23AM (#39756713)
They're mainly trying to issue certs for public facing websites. If your service is in a position where someone can install an alternative root CA on your webserver, you're already pwned!
Re:New Intermediate Certificates by gkbrk on Tuesday March 19, 2024 @11:27AM (#39756770)
The threat model isn't "someone will install stuff on our servers". The threat model is "some user will look at the network traffic and see all the personal data we're hoovering from their phone".

Sometimes also "some user will look at the network traffic and notice that we have 0 server-side security".

Re:New Intermediate Certificates by kobalsky on Tuesday March 19, 2024 @11:38AM (#39756884)
Also, some companies[1] have been caught being naughty and installing CA certs which allowed third parties to MiTM local connections. This was a major issue and, at least in my case, what drived many to pin certs at the time.

[1] https://en.wikipedia.org/wiki/Superfish#Lenovo_security_inci...

Re:New Intermediate Certificates by Sohcahtoa82 on Tuesday March 19, 2024 @01:19PM (#39758065)
> The threat model is "some user will look at the network traffic and see all the personal data we're hoovering from their phone".

If you're talking about someone wanting to examine an app they have installed, there's nothing a developer can do about it.

There are frameworks like Frida that allow you to disable pinning or disable certificate verification entirely in an app. I did this myself once while reverse-engineer the API that an app was using. Configured my phone's system proxy to point to Burp Suite, then used Frida to disable certificate verification in the app so it'd let Burp do its MitM magic, and I could see all the app's network traffic.

Re:New Intermediate Certificates by vbezhenar on Tuesday March 19, 2024 @11:37AM (#39756876)
You must pin your own key, not certbot key.
Re:New Intermediate Certificates by aaomidi on Tuesday March 19, 2024 @11:41AM (#39756920)
What does this mean?
Re:New Intermediate Certificates by phasmantistes on Tuesday March 19, 2024 @11:45AM (#39756959)
I believe they are using "certbot" to mean "Let's Encrypt", in which case their advice is sound -- if you truly have to pin a key, pin your own end-entity cert's key, not the CA's key.
Re:New Intermediate Certificates by pixl97 on Tuesday March 19, 2024 @11:48AM (#39757000)
I'm assuming out of the 3 levels that are on most certificates

Root / Intermediate / Server, you only pin server certificates and not the levels above them.

Re:New Intermediate Certificates by tialaramex on Tuesday March 19, 2024 @12:42PM (#39757601)
These three "levels" are the minimum that would work in the Web PKI and thus usual (for public services).

The reasoning goes like this. Roots represent entities you trust (or in most cases, which are trusted on your behalf by some entity that's thought about what constitutes trustworthiness e.g. your browser vendor, OS vendor, or maybe your workplace). Accordingly the root "certificates" aren't really certifying anything, the X.509 certificate format is a convenient structure to put this data in, but unlike most certificates they don't represent a certification of anything by anybody, in human terms these documents tend to say e.g. "We, ISRG claim that we are ISRG and we're a Certificate Authority and we're signing this document to say so". OK, but anybody could make such a self-signed document, there's no way to "authenticate" them, you must have decided to trust some of these claims (or as I said, your vendor probably did for you)

The private key corresponding to a root is therefore very precious, billions of people trust signatures made with this key so it's a huge deal if it's stolen/ compromised. As a result we require that these keys aren't online, they're typically manifested as a small physical object (a "hardware security module") locked in a safe, the idea is that the key itself cannot easily leave the object, so by locking it in a safe we've protected the key.

But since these roots can't be online, we can't make certificate with them online, so a simple system of the sort you might build at work, with a CA root that directly issues certificates, isn't possible. This is where the Intermediates come in.

A ceremony is conducted, with third party auditors and senior personnel from the CA to watch as the Root is used to sign one or (as here for ISRG) a handful of Intermediate certificates. These documents can usefully be verified because they were signed with the Root's keys, if we trust a Root we can check that indeed it has signed this Intermediate. The CA is also required to have a technology so that it can (in a reasonable amount of time) revoke an intermediate if for example it was compromised or stolen.

Unlike a root, the Intermediates can then be kept online (at ISRG only some are used this way, others are just backups against the unexpected) and used to sign End Entity (what you've called Server) certificates in real time shortly after they are ordered, once the appropriate checks have been carried out. HSMs are used again, but in this case they're attached to a server where the issuance software is running rather than locked in a safe.

This post is about ISRG using new Intermediates.

Re:New Intermediate Certificates by vbezhenar on Tuesday March 19, 2024 @03:30PM (#39759388)
By certbot key I mean letsencrypt key, sorry for confusion.

That means that you need to hard-code the public key of the certificate that you're using in the client application. That certificate might be signed by letsencrypt or any other CA, it doesn't matter. If you're pinning CA key, you put trust into entity which you don't control. They can replace any of their keys. They can stop signing your certificate. It makes little sense to pin CA certificates.

Just make sure to have a plan for key rotation in case you need to revoke your key.

Re:New Intermediate Certificates by dist-epoch on Tuesday March 19, 2024 @11:29AM (#39756791)
It also prevents your employer from snooping on you if you open Signal/WhatsApp/Facebook/... on your work computer.

Of course, you might consider that a legitimate employer thing to do.

Re:New Intermediate Certificates by withinboredom on Tuesday March 19, 2024 @11:28AM (#39756773)
Maybe get a proper certificate designed for such things instead of using what amounts to a corporate-funded public service.
Re:New Intermediate Certificates by djbusby on Tuesday March 19, 2024 @11:41AM (#39756919)
What is improper about them? Works for 100% of my use case, well known, trusted, compatible.
Re:New Intermediate Certificates by aaomidi on Tuesday March 19, 2024 @11:42AM (#39756929)
Let's Encrypt, and WebPKI in general is meant for public consumption of certificates.

I think what this means is, if you're needing some stricter limitations on these, use private PKI within your own devices. E.g. for Machine to Machine comms.

Re:New Intermediate Certificates by withinboredom on Tuesday March 19, 2024 @01:51PM (#39758440)
What I mean is, if you want to pin certificates, purchase a certificate that expires in years, not months.
Re:New Intermediate Certificates by vbezhenar on Tuesday March 19, 2024 @03:39PM (#39759476)
You pin public key, not certificate. You can keep your private key for whatever period you want and issue multiple certificates with the same public key.
Re:New Intermediate Certificates by tptacek on Tuesday March 19, 2024 @01:15PM (#39758014)
Indeed, many of the vendors who will sell you proper certificates will also sell you anti-tiger rocks, which you should also consider procuring.
Re:New Intermediate Certificates by duskwuff on Tuesday March 19, 2024 @01:29PM (#39758189)
Some of these anti-tiger rocks even come with an anti-tiger warranty. If you can confirm that you were attacked by a tiger and the rock failed to protect you, you can make a claim for the price of a new rock.
Re:New Intermediate Certificates by pimterry on Tuesday March 19, 2024 @12:54PM (#39757752)
> This is pretty commonly used by apps to make them harder to reverse engineer. It prevents you from installing your own cert in the system's trust store and then using that to man-in-the-middle the app's communication with its backend.

As an aside - this isn't generally very effective or worthwhile nowadays.

Anybody doing reverse engineering at a level where they're installing their own system certificates & intercepting traffic can easily run many off the shelf scripts (Frida scripts, Objection, apk-mitm, etc) which will disable certificate pinning like this automatically. This takes _seconds_ to disable and is standard practice.

Even doing so manually is not especially difficult - the pin is fairly predictable value, and a reverse engineer can access all content of the mobile app, so can easily search for and just replace it before installation.

From a reverse engineering perspective, there's little-to-no value in purely client-side protections like this. You're not increasing the reverse engineering difficulty significantly and you do create many practical problems with e.g. certificate rotation in future.

OTOH there is an interesting case to be made for certificate pinning as a protection for users being unknowingly MitM'd. That's a different scenario that may well be worth defending against, but due to the many downsides of cert pinning, using certificate transparency to mitigate this is strongly preferable nowadays.

Re:New Intermediate Certificates by xg15 on Tuesday March 19, 2024 @06:38PM (#39761048)
Mostly agreed - I think the one difference is that to disable the pin, you have to modify and sideload the APK, which a non-rooted phone may not permit you to do. (Or your modified apk might not have access to the app's local data, security keys, etc).

In contrast, with a non-pinned app, you can monitor traffic of that app as it is installed right now.