There's a reply from Ben Wilson (Mozilla) further down in the thread / the next day stating that Firefox as a client is not affected by the security issue (OCSP responses signed by these intermediate CAs would be rejected), and that Mozilla is not planning on enforcing the 7 day deadline for revocation of these intermediate certificates. The CAs will still need to replace these intermediate CAs, but with a more gradual timeline.

https://www.mail-archive.com/[email protected]...

> We are concerned that revoking these impacted intermediate certificates within 7 days could cause more damage to the ecosystem than is warranted for this particular problem. Therefore, Mozilla does not plan to hold CAs to the BR requirement to revoke these certificates within 7 days. However, an additional Incident Report for delayed revocation will still be required, as per our documented process[2]. We want to work with CAs to identify a path forward, which includes determining a reasonable timeline and approach to replacing the certificates that incorrectly have the id-kp-OCSPSigning EKU (and performing key destruction for them).

Of course, the point is that by bylaws the CAs themselves agreed to, in their own BRs, they're required to revoke the certificates within 7 days. It's a SHALL requirement.

The CAs can't have it both ways: a BR balloting process that they rely on for moral authority when disputing that the majority of deployed browsers have added new security requirements (like shorter-lived certificates), and BRs that they ignore when they screw up.

Can't they have it however they want as long as the vendors that matter go along with it? I feel like Ryan is sort of pointing that out, that an excuse of "oh well we needed to support vendor X" gives them carte blanche. It's not like customers are going to knock down their doors for not following BRs. At the end of the day, the biggest vendors are where their bread gets buttered.

If Mozilla isn't the majority browser vendor, who cares what they insist on? And if all the CAs band together and say, sorry losers, we're gonna keep doing things our way, what are the browsers gonna do? Cut all their users off from the internet "because principles"? Apple is playing a dangerous game that I don't think will work out in different circumstances. They can't hide behind "protecting users" if their users end up unable to access the internet securely.

We got into this mess because we wanted organizational independence and distributed trust, without considering what internal conflicts would mean to the end users. I'm going to call it and say that within a decade, you'll have to pick which CA you want to trust at browser install time (though you can guess which CA will be the default on which devices).

This is a weird comment. Leave aside that anything the Mozilla root program does will, if history is a guide, likely be backed immediately by Google. Minority browser or not, if your CA isn't trusted by Mozilla, you will lose all your customers. Nobody is going to do business with the CA that offers "the majority of browser users" when the rest of the CAs offer all the browser users. The CAs have almost no cards to play here.

> And if all the CAs band together and say, sorry losers, we're gonna keep doing things our way, what are the browsers gonna do?

For one thing you should consider that several of the companies that make browsers also operate a Certificate Authority. For example Google's GTS is represented in that m.d.s.policy thread by Ryan Hurst (whereas Ryan Sleevi is there mostly choosing not to put on his Chromium Web PKI hat). Microsoft likewise controls trusted roots. Apple does operate its own PKI but is not presently broadly trusted, though if it felt the need I'm sure they have people.

Also the most popular (for this purpose) Certificate Authority is ISRG's Let's Encrypt and they've got no reason not to co-operate. The Web PKI is all they do anyway.

This is often portrayed as though browsers are obliged to either distrust everything instantly or allow CAs to do whatever they like, but neither of those is realistic. The major trust stores all already have imposed constraints short of distrust.

You bring up Apple as an example. Unless I've missed it somewhere Apple never announced their 398 day limit as a matter of issuance policy it's simply a fact that Safari and the Mac operating system won't trust new certificates with longer lifespans after a set date. So if one or more CAs decided not to co-operate, nothing happens at first. Nothing whatsoever.

Then, gradually, a few subscribers buy (or renew) 2 year certificates, and these new certificates don't work in Safari. Some of these subscribers will call customer services at the CA where they purchased the certificate. Why doesn't it work in Safari? How can they fix this?

What does the CA say? "We intentionally sold you a product that won't work. Ha ha ha, it's a funny joke, we have your money and you've got useless garbage" ? Maybe they instead try to blame Apple. Apple will point out that the CA knew this wouldn't work and suggest the subscriber seek a refund.

The subscriber demands a refund. The CA is now losing money and it is seeing negative reputational impact. Somebody threatens to sue. It is not a good day to be the CA.

At the subscriber's offices, an IT person has a brainwave and switches to a provider that isn't deliberately disobeying. The web site is back working. Champagne all round.

To achieve the desired robustness/ reliability the Web PKI is structured in a way that makes any individual Certificate Authority expendable. As a subscriber this means you should plan for your CA going away with at most a few days notice. Most people won't do that. Too bad, individually you're expendable too.

Apple did actually make the limit a matter of their policy. Start date is September 1, technical enforcement comes 'later', but the requirement is going to be part of their policy. https://lists.cabforum.org/pipermail/public/2020-April/01494... Plus, it's not just Safari - it's the TLS stack in macOS/iOS etc. - Apple's KB article mentions nothing of Safari, but just like their enforcement of CT, it's basically OS-wide.

I don't think this is how it would (could?) play out. The CA would say, "Your Apple software is buggy, because nothing else has this weird limit. You can buy a 1 year cert from us to fix your issue. But because you already bought a 2 year cert, and Apple is being a jerk to you, here's an additional 1 year cert. Tell Apple they suck for making you jump through this hoop." And Apple would drone on about security or privacy or whatever being worth the customer now having to do more work. If nobody else played ball, it really would be just Apple annoying everybody. For reasons I can't yet fathom, the rest of the CAs and browsers seemed to chicken out instead of calling their bluff.

That’s about what I expected. It doesn’t make sense for any parties involved to enforce the 7 day deadline, including users. This is a reasonable solution.

It's just slightly disappointing that the CAs and browsers have agreed to operate under a policy that requires the CAs to revoke mis-issued subordinate CAs within 7 days (CA/B Forum BR 4.9.1.2), but that's not actually feasible in practice - at least not at this scale. Certificate issuance would need to be automated enough that it would be feasible to actually replace all certificates issued by the affected CAs within a 7-day period.

Let's Encrypt represents the state of the art in terms of certificate automation, but last time they had an (impending) mass-revocation event, it turned out that even the ACME protocol / client implementations didn't really have any concept of an automated "this certificate is about to be revoked, please re-issue" process. As a result of that event, certbot at least got support for triggering renewal after revocation: https://github.com/certbot/certbot/issues/1028#issuecomment-... -> https://github.com/certbot/certbot/pull/7829

I think it's not so much that it isn't feasible as that it doesn't serve any useful purpose at least to Mozilla.

The Firefox out-of-band revocation mechanism (OneCRL) certainly could revoke all these intermediates but Firefox isn't vulnerable to a problem here, so there's no obvious upside to doing that and it's disruptive.

Maybe I'm a bit uninformed but I feel that the whole certification system is quite convoluted and confusing. Couldn't there be a simpler solution of solving the simple trust problem that a user has over the authenticity of a website?

There are technical issues involved and the system has evolved over the years, but part of the issue is that certificate companies make a vast amount of money doing very little and can spend a bunch of money resisting any change to the system that would cut them out of the loop.

IMO, DANE might make sense if DNSSEC wasn't such a mess, although it is a very similar group of parasitic companies involved in DNS. In general, alternative name systems (such as the GNU Name System) could also potentially replace the certificate system and many name and certificate issues are related. Many of the hardest technical issues around certificates relate to revocation and the demonstrated inability of almost anyone to secure anything.

Other options that make a lot of sense in many ways would have govenments or banks involved in identity in a direct way. This is resisted for a varity of reasons.

> if DNSSEC wasn't such a mess

I hear this a lot, but in my experience (managing c. 1000 DNS zones all with DNSSEC enabled, using a strictly DNSSEC-validating resolver for >5 years, and having built DNSSEC infrastructure for DNS hosting providers), it is both reasonably well designed and generally quite well implemented. What is the mess that you perceive?

> What is the mess that you perceive?

Not GP, but the mess is near-zero clients and few recursive resolvers are actually doing DNSSEC validation in practice, after 20 years of deployment.

It’s like IPv6.

Also I believe most active ZSKs are actually held and managed by the larger DNS providers on their customers’ behalf. This leads to very little assurance improvement over unsigned records, as credentials to update a web form is all that is needed to “sign” records. There are no real key management requirements for ZSKs as there are with browser CAs.

The only additional assurance provided by a DNSSEC response is that there was likely no MITM between the authoritative server and validating resolver. Which is something, but that problem is more easily and completely solved by DoH which adds privacy as well as authenticity.

The bigger problem is that none of the browsers support it, or intend to support it. DoH could change that (ironically, the DNSSEC crowd opposes the one DNS change that could offer any chance of viability for DNSSEC), but it's unlikely to.

Well, if the system resolver is doing DNSSEC validation, the browser doesn't need to support it. That's how I've had my system set up for years. Unfortunately, macOS and Windows (other than Server) still don't have any DNSSEC support as far as I'm aware. Support is built in to systemd-resolved now though, which I believe is the default resolver on various Linuxes, and unbound is of course available in all major distros.

If the goal is to have DNSSEC replace the CAs, then you need DANE, and for DANE to work, browsers have to support it.

That can indeed be a goal, but DNSSEC is useful even without DANE.

(a) No it can't be, and (b) that's not what this thread was about.

Interestingly the GNU Name System IETF draft was just officially filed yesterday: https://tools.ietf.org/id/draft-schanzen-gns-01.html

I'm afraid the solution cannot be simple, not only because the trust problem is not so simple, but because the nature of the problem is political, not technical. And so the solution must be political (CA/B forum, ballots, convoluted documents, processes for misbehaving entities).

>The author goes on to explain that revocation of the affected certificates is insufficient, because they could be used to effectively reverse their own revocation at any point in the future. Instead, it must be proven that all copies of the keys have been destroyed. That’s quite an undertaking.

How would this be verified? Presumably the keys are stored on HSMs, but you can I'm not sure how you can prove that you didn't make a backup of the key.

It is largely impossible to fully prove. CAs are supposed to keep detailed records of any issuing keys and what was called for was specifically "witnessed Key Destruction Reports" which involves third party independent confirmation of destruction of documented keys.

In the event that a key with a Key Destruction Report shows up again, the responsible party for that key will have shown unacceptable negligence and will potentially be subject to the exclusion of their keys as a valid certificate signer.

A lot of these companies core businesses rely on remaining in a position to sign certificates so it is in their best interest to protect that privilege by following the documentation requirements, and properly destroy their keys. It's effectively a pretty good stick.

It’s impossible to prevent a truly dedicated malicious actor from doing so, but enforcement through both policy and independent auditors — and quality of response to security incidents — provide several layers of defenses against this scenario. (As with all things, a perfect defense is ultimately impossible, but they put in a lot of effort to get a lot of nines of certainty.)

In principle, you would hope a CA would be able to precisely account for the number of backup copies of their private key.

In practice, of course, that doesn't mean every one of them will have done. There's 293 of them, after all.

The point is that you can't verify it. Sometimes HN humor is too dry for its own good.

What are the state-actor-level attack implications of this? Before this was revealed, a party that compromised (or was able to be issued) a certificate for a website could be reasonably likely to be detected and have that certificate revoked if they used it for large-scale MITM or redirection. But now, if the actor were to also compromise any one of these sub-CAs before the key was deleted, could they permanently be in a position to be able to unilaterally reverse any such revocations, effectively giving them carte blanche to begin a campaign of compromising websites in earnest with the knowledge that their attacks would now be "sticky?"

What would the recourse be here if one of those keys were to be compromised, or even if there was reason to believe one might have been? Would the entire CA-level trust chain need to be distrusted, requiring re-issuance of all certificates on that chain?

Yes and almost-yes - you don't need to destroy the root CA itself, but you need to physically destroy the keys for any affected subordinate CAs. (Revoking them isn't sufficient, because they can just un-revoke themselves.) From the post: "... the degree of this issue likely also justifies requiring witnessed Key Destruction Reports, in order to preserve the integrity of the issuer of these certificates (which may include the CA's root)."

The "good" news is that most people haven't really been treating revocation (and OCSP) as a reliable mechanism. The major browsers all have out-of-band mechanisms for revoking known-malicious certs via something equivalent to the software update channel, which bypasses reliance on the CA infrastructure. If there's a large-scale attack, the relevant cert/CA will probably be disabled through that mechanism. And most of the smaller clients don't even bother with revocation checking at all (e.g., I'm pretty sure that on an average Linux system, things like curl or "import requests" do no revocation checking) so there's no point in undoing revocation if you're trying to attack one of those systems.

> e.g., I'm pretty sure that on an average Linux system, things like curl or "import requests" do no revocation checking

This is correct. Even where there's some provision for checking, it's usually a mechanism where you can supply a CRL (Certificate Revocation List, a signed and dated document that says which certificates were revoked). CRLs are practical for a small private CA but they make no sense at scale. Let's Encrypt doesn't even have CRLs because they'd be enormous.

To be fair 10-15 years ago there's a good chance that average Linux system has a set of CA roots which hasn't been updated in a decade, and most such clients aren't actually checking even CN let alone SANs so bad guys don't need a google.com certificate (or whatever) they can just get themselves a real certificate for actual-bad-guys.example and the client won't check the name matches anyway.

Having sadly worked with plenty of developers whose first reaction to an invalid certificate error is to Google how to disable the checking, I don't think you need to go back 10-15 years.

Interesting. I do imagine there are a nonzero number of systems that do revocation checking without out-of-band mechanisms, and that those would become susceptible. But all such systems are likely vulnerable to unpatched zero-days anyways, so this may not make things any worse than they already are :/

> What are the state-actor-level attack implications of this? Before this was revealed, a party that compromised (or was able to be issued) a certificate for a website could be reasonably likely to be detected and have that certificate revoked

This is a convenient fiction. CA system never protected anyone against state actors. Never did, never will. Subverting a single CA is enough to compromise entire system. And there are hundreds of them.

Security is always grounded in knowledge and physical control — understanding and exercising your capabilities to preserve them. A blind, deaf and fully paralysed person can't be expected to safeguard their own physical security, and neither can an average user — their TLS security. Especially against state actors. More so, when the parties they have to rely on are commercial enterprises whose entire existence revolves around getting paid to issue certificates.

This is all confusing to me but I've been having certificate issues today and it seems like this could be related. It's a weird coincidence if not!

Basically on Chrome one of my sites is throwing:

"NETT::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED"

for most users but not all, even though they're all on Chrome. It seems to work fine in other browsers.

https://transparencyreport.google.com/https/certificates

When I check my domain here it seems like I have got the transparency certificate so I shouldn't be getting this error.

Is this related to what you're talking about? I would really appreciate any help. I'm using https://letsencrypt.org/ for the cert.

Does your cert have an SCT? It would be strange for a Let's Encrypt cert to be missing it but certainly possible. Try running (replace both example.coms with your domain name)

    openssl s_client -connect example.com:443 -servername example.com </dev/null | openssl x509 -noout -text

which should print an SCT extension at the end - my version displays it by numeric identifier "1.3.6.1.4.1.11129.2.4.2" but maybe newer versions display it by name.

Alternatively, I think you might able to go to https://www.ssllabs.com/ssltest/ and see if your cert has "Certificate Transparency: Yes", but I'm not sure exactly what that means.

Anyway, I don't think this is related, the question at hand is about OCSP, which is a different mechanism from Certificate Transparency. (Arguably Certificate Transparency is a replacement for revocation in general being flawed in practice for many reasons, but they're different mechanisms.)

It's a weird coincidence for you but for everybody else it's to be expected as there are dozens or hundreds of people having issues every day.

It's extremely unlikely to have anything to do with this incident.

You should obtain a copy of the certificate which triggers NET:ERR_CERTIFICATE_TRANSPARENCY_REQUIRED and take a look at that. There's an excellent chance there's something else even more obvious wrong (from your point of view as a human) but Chrome decided to focus on the lack of trustworthy SCTs.

My instinct would be that it's likely a middle box (e.g. "anti-virus software" on a PC can install itself to snoop on all HTTPS sites, or a corporate "data loss prevention" proxy or that sort of thing) and the bogus certificate will likely make that pretty obvious if you examine it.

I think it's an interplay between system clock skew, Chromium's SCT validation implementation, and (very) recently issued certificates (which are backdated by 1 hour).

It's a bit of a heisenbug but it's occasionally reported on the Let's Encrypt forums. It always goes away for the reporters just by waiting a little bit.

It would be really nice if a user who runs into this could generate a Chromium event log which would hopefully include the SCT events (chrome://net-internals).

Thanks! It does seem to have gone away today. Very strange.

That is probably not related.

If you run SSLLabs against your host name, does it say “Certificate Transparency: Yes” or No?

https://www.ssllabs.com/ssltest/analyze.html?d=your-hostname

Thanks. It says "yes".

"...revocation of the affected certificates is insufficient... Instead, it must be proven that all copies of the keys have been destroyed."

Isn't the main reason you would want to revoke keys because they were disclosed, making it impossible to destroy all copies?

Normally, yes. This is not a normal circumstance though. In this scenario, the misissued intermediates effectively have sudo access to cancel a revocation issued by the parent CA. This is equivalent to being told that a non-root user could cancel a userdel command run by root. For policy compliance, the intermediates have to revoke their certificates — but since the intermediates can immediately un-revoke themselves, proof of key destruction is necessary as well to ensure that they cannot.

Further, if I can destroy all keys do I even need to revoke the cert? (Honest question, I’m actually not sure)

Revocation is required by policy, so the question is technically moot. It’s generally good practice to generate and publish a revocation prior to destroying a private key, though.

To provide an analogy in the context of PGP keys, if an attacker somehow finds a backup of your revoked and destroyed private key someday, they will have trouble using it because your revocation will be public and on record.

Flagging for a title change, as the revocation deadline was proposed by a single individual on a mailing list, and there’s no evidence from further discussion in that thread that a consensus has been reached. I would not expect all these certificates to be revoked within 7 days.

That "single individual on a mailing list" is Ryan Sleevi, who is basically the voice of Google in all matters to do with TLS PKI, and as I understand it is not working with a "opinions my own" cop-out in this context. It's not some rando

In addition to his role with Google/Chrome, he is also a peer of the Mozilla CA module and has a lot of influence in Mozilla's policies as well.

It’s still one person. There needs to be a consensus, and that consensus hasn’t been reached. As has already been mentioned in other comments, Mozilla said they wouldn’t enforce the 7-day deadline.

Flagging doesn’t result in title changes, it just terminates the post. Emailing the mods using the footer Contact link does, though.

Thanks. It’s a moot point now, but I’ll note that for the future.

I changed the title. Although I heard from someone involved that the intermediates really should be revoked in 7 days. Let's wait and see.

They definitely should be—that’s what the author is claiming is mandated, and it would make sense. However, I’m a bit skeptical about browsers being able to enforce that timeline here.

Also, given that the underlying cause appears to be ignorance, it would be prudent to take things slow and ensure that this doesn’t happen again. As I said before, the damage is already done—revoking appears to be insufficient here.

If this does actually happen within 7 days, though, I will be thoroughly impressed.

It could be considered a tacit warning that browsers may choose to mistrust the impacted subCAs in the near future. I don’t know the specifics, but I assume they can revoke for non-compliance using in-browser mechanisms without depending on the revocation process.

EDIT: Mozilla’s reply: https://news.ycombinator.com/item?id=23748561

You don't have to trust Sleevi (though: you always should); you can just read the BRs. The revocation requirement is in this case black-letter SHALL.

https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-...