News for hackers. Stuff that matters.
Re:Is HTTP Public Key Pinning Dead? by eganist on Tuesday September 06, 2016 @06:52AM (#12435002)
I figured interesting words would probably start to trickle out about HPKP after the last month or so of talks. Opinions are split on how much damage RansomPKP and other hostile pinning attacks can do. I'm inclined to believe that it's a big enough risk to warrant some action (such as more diligence by free certificate authorities -- the crux of the talk was rekeying at up to 20x per week enabling the attack). Others, notably Mozilla, are taking the wait-and-see approach. Google and Let's Encrypt were the most resistant to changing anything at all.

Anyway, Scott Helme's writeup did a quick and solid job of summarizing a lot of it. It's linked from the Qualys piece, but here it is if you'd rather get to it directly: https://scotthelme.co.uk/using-security-features-to-do-bad-t...

Full Disclosure: I and buu700 (https://news.ycombinator.com/user?id=buu700) did the AppSec Glory talk at Black Hat / DEF CON and developed the RansomPKP attack pattern.

Edit for Fuller Disclosure plus added thoughts: I'd rather not see HPKP die because we've done some cool stuff with it over at Cyph. I'd rather see HPKP be refined with extra requirements, possibly some sort of tie-in with CAA (that little-known standard for enforcing which CAs can issue certs for your domain).

Speaking of CAA, we did a show of hands in both the BlackHat and DEF CON versions of the AppSec Glory talk. Five hands across thousands of attendees. That's it.

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @08:18AM (#12435383)
I don't think blocking rapid re-keying on the CA side is going to do much good to prevent this kind of thing. It would help with the pattern described in RansomPKP, but that's not the only option available to attackers. The web server does not need direct access to the (maliciously added) private key - a separate service under the attacker's control that holds the private key and signs stuff for the web server would be sufficient (this is basically what CloudFlare does with Keyless SSL[1]).

I see some potential for CAA to help prevent these attacks. Limiting issuance to a small number of CAs would help, especially if those CAs are ones you have a business relationship with and have agreed to some sort of out-of-band confirmation process for all certificate requests (I believe this is fairly common for high-risk domains). There's been some discussion about account key pinning via CAA in the ACME WG. Account keys could be kept separate from the web server (possibly even in a HSM), so this could also help mitigate this problem.

That being said, none of that is going to do any good while checking CAA records is optional for CAs, not to mention that just about no one knows it even exists (even in the infosec community, as you have demonstrated). Hopefully we'll see some movement on that front in light of recent mis-issuance reports.

[1]: https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-tec...

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @12:28PM (#12437182)
> CAA (that little-known standard for enforcing which CAs can issue certs for your domain)

I'm sure the CA's absolutely LOVE this - customers are signaling that they will stay with CA 'X'.

Given 'vendor lockin' or 'guaranteed recurring revenue', cert prices would drop thru the floor for 1st year only for renewals to increase.

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @12:43PM (#12437298)
CAA is just a DNS record that can be changed at any time, so there's no vendor lock-in.

CAs that implement CAA request that record and check whether they're permitted to issue the certificate; if they're not, they should refuse issuance.

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @01:04PM (#12437453)
1) CAA tries to address the threat of attacker getting a cert issued for your domain - to carry out such an attack inherently requires taking over your DNS record or your domain account - if they can do this then they can also reconfigure your CAA record.

2) CAA requires the CA to perform additional operation (retrieve and check that DNS record) - not all of the 300-600+ CA's will do this - all it takes is 1 CA and the attacker will get his fraudulent cert from that CA.

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @01:23PM (#12437622)
1) No, it is not necessary to have full control over your domain's DNS to obtain a certificate. There are various domain validation mechanisms. Having full control over the HTTP server (as mentioned in the article) would be sufficient, as would having access to certain email addresses associated with your domain.

2) I don't know why you're telling me this - it's not related to either your original post, or my reply. I'm quite aware that CAA is only really effective if it is mandatory (in fact I've mentioned it in a sibling comment a couple of hours ago).

Re:Is HTTP Public Key Pinning Dead? by notatoad on Tuesday September 06, 2016 @01:14PM (#12437546)
>CAA requires the CA to perform additional operation (retrieve and check that DNS record) - not all of the 300-600+ CA's will do this - all it takes is 1 CA and the attacker will get his fraudulent cert from that CA.

CAA seems like another good step that the browsers can demand after a CA fucks up - like google has mandated that some CAs publish Certificate Transparency logs after they've mis-issued a cert, they should also be requiring CAA on threat of being revoked from the browser's trust lists.

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @01:22PM (#12437600)
Yeah - in theory, but in reality the browsers are reluctant because of all the other sites that a CA has issued certs for - no browser wants to be singled out for blocking some other non-related sites.
Re:Is HTTP Public Key Pinning Dead? by ivanr on Tuesday September 06, 2016 @01:09PM (#12437496)
For what it's worth, my impression is that CAA will be mandated by the CA/Browser forum at some point. But, indeed, that's the main weakness of CAA—it requires that substantially all CAs support it.
Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @01:19PM (#12437576)
There's too many paying entities to appease - not just hundreds of CA's but various browser vendors as well. Either MUSTs will be changed to SHOULDs - or fragmentation of the CA/Browser body itself.

Look no further than at some of the past transgressions browsers let CA's get away with.

Re:Is HTTP Public Key Pinning Dead? by ivanr on Tuesday September 06, 2016 @12:37PM (#12437250)
Maybe, but there's no memory effect associated with CAA and you can change your configuration it any time. You're not actually locked in.
Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @12:46PM (#12437325)
> with CAA and you can change your configuration it any time

so too could an attacker. If an attacker is able to set HPKP, then they could just as easily reconfigure CAA to a CA that issues them a cert for your domain.

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @12:51PM (#12437363)
The attack described in the article (web server compromise) would not necessarily give you access to the DNS of your domain (that's putting a lot of eggs in one basket).

It's certainly an effective defense-in-depth mechanism.

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @01:12PM (#12437526)
> It's certainly an effective defense-in-depth mechanism.

1) We've seen time and again that complexity is the enemy of security. Generally, the more moving parts, the more likely for another flaw. This is less defense-in-depth than it is added complexity. The attackers have time on their side to figure out where the next hole is.

2) The OP suggests that HPKP failed because of practical reasons (domain admins are scared to death of getting it wrong and bricking their site) - things like CAA only add to the complexity.

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @01:37PM (#12437718)
1) I don't see how it can possibly be worse than the status quo. CAs already rely on DNS for domain validation. This is just another DNS query, with the reply being a whitelist of permitted CAs, not a replacement for domain validation. If the CA fails to follow the whitelist, it's not worse than a CA that does not implement CAA. Without pointing out evidence that shows how this addition could make things worse, I don't think this is a good argument against CAA.

2) CAA was introduced in this discussion as a possible solution to the RansomPKP problem. It is not a requirement for HPKP and the only effect on HPKP it would have is to actually reduce the risk associated with that attack.

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @01:56PM (#12437886)
> CAs already rely on DNS for domain validation

Not all - only a subset of certs issued

> If the CA fails to follow the whitelist, it's not worse than a CA that does not implement CAA. Without pointing out evidence that shows how this addition could make things worse, I don't think this is a good argument against CAA.

because the USER can't tell. The user is under the impression of increased security (because upgrading to browser version X.Y said it now supports CAA) yet the user doesn't know which cert was issued by a CA that checked CAA. And you can't add yet another indicator to the UI because the user is already numb from just the certificate itself.

> actually reduce the risk associated with that attack

and as I tried to point out to you, it's ineffective against the threat of an attacker gaining access

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @02:15PM (#12438023)
> Not all - only a subset of certs issued

I'm not sure I follow. My point is that if you're arguing that a DNS query is some kind of added complexity, then I have bad news for you, because DNS queries are already a part of all domain validation methods (whether it is email, HTTP, DNS, etc.)

> because the USER can't tell. The user is under the impression of increased security (because upgrading to browser version X.Y said it now supports CAA) yet the user doesn't know which cert was issued by a CA that checked CAA. And you can't add yet another indicator to the UI because the user is already numb from just the certificate itself.

This has nothing to do with browsers. It's a mechanism to improve the CA domain validation process. It can help mitigate certain vulnerabilities in the domain validation process. For example, WoSign recently mis-issued certificates for github.com due to a bug in their domain validation system. If they had used CAA, and github.com had a CAA record indicating WoSign is not permitted to issue certificates for that domain, it's quite possible that the certificate would not have been issued.

CAA is not a replacement for HPKP, and no one is arguing that. It's an useful defense-in-depth mechanism that could help prevent mis-issuance in many cases, but it's not (and never has been advertised as) a solution to the problem of a fully-compromised CA. In addition to that, as has been mentioned in this thread, it might be useful in mitigating RansomPKP to a certain degree in the future.

> and as I tried to point out to you, it's ineffective against the threat of an attacker gaining access

No, you have not pointed that out. I have pointed out that you need access to the DNS in order to change (and bypass) CAA records, whereas you only need control over the web server in order to obtain a certificate that can be used to ransom a domain.

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @02:46PM (#12438284)
> I'm not sure I follow. My point is that if you're arguing that a DNS query is some kind of added complexity, then I have bad news for you, because DNS queries are already a part of all domain validation methods (whether it is email, HTTP, DNS, etc.)

We're talking about CA's issuing certs - DNS queries for email, http are outside of CA cert issuance.

> This has nothing to do with browsers. It's a mechanism to improve the CA domain validation process.

The browsers need to decide when to trust a cert - and if/when CAA becomes involved in cert issuance, then I suggest this DOES have something to do with browsers. Furthermore, you're suggesting co-existence between CA's that check CAA and CA's that don't - which implies that either the browser or user has to make a determination of whether to trust.

But we're talking past each other. So using your example, think of it like this, how will you as a user know when you visit github that another WoSign hasn't happened? ...or rather - how would a LAYPERSON know they are secure? Github might detect it - but how do ordinary USERS?

> If they had used CAA, and github.com had a CAA record indicating WoSign is not permitted to issue certificates for that domain, it's quite possible that the certificate would not have been issued.

that (wrongly) assumes ALL CA's trusted by EVERY browser will perform CAA check before issuing a cert for github.

> CAA is not a replacement for HPKP, and no one is arguing that. It's an useful defense-in-depth mechanism that could help prevent mis-issuance in many cases, but it's not (and never has been advertised as) a solution to the problem of a fully-compromised CA.

The problem is not so much "fully-compromised CA" as it is we're trusting a whole pile of CA's and not all them behave the same.

> you need access to the DNS in order to change (and bypass) CAA records, whereas you only need control over the web server in order to obtain a certificate that can be used to ransom a domain

Any attacker that can gain access to web site admin credentials can also get the DNS credentials.

> CAA is not a replacement for HPKP, and no one is arguing that. It's an useful defense-in-depth mechanism that could help prevent mis-issuance in many cases

You're using defense-in-depth again. Replace "-in-depth" with "-added-complexity". Here's a directly relevant example - HPKP arose from deficiencies in static pinning - which arose from deficiencies of CA/SSL ecosystem - which arose from deficiencies of plaintext traffic. It's not adding security "-in-depth" when all you're doing is resolving deficiencies in existing deployed solutions. The "depth" is single-level, not multiple.

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @03:08PM (#12438459)
> We're talking about CA's issuing certs - DNS queries for email, http are outside of CA cert issuance.

Still doesn't make any sense to me. Whether a CA performs a DNS query in order to do domain validation via email, http or to check a CAA record doesn't matter.

> The browsers need to decide when to trust a cert - and if/when CAA becomes involved in cert issuance, then I suggest this DOES have something to do with browsers.

The domains that a CA "vouches" for are part of the certificate. Browsers tell CAs how to determine domain ownership (by telling them to follow the Baseline Requirements). Implementing CAA means making the CAA check a mandatory component of these requirements. In other words, this would be just another step and if the domain appears on a certificate, the CA would indicate that the CAA check was successful. Browsers are the ones who would mandate this, yes, but there would be no other change necessary.

> Furthermore, you're suggesting co-existence between CA's that check CAA and CA's that don't - which implies that either the browser or user has to make a determination of whether to trust.

I'm not suggesting that. I'm suggesting making CAA mandatory. The scenario you're describing is the status quo, by the way: Some CAs implement CAA (Let's Encrypt, DigiCert); others (the majority) don't.

> But we're talking past each other. So using your example, think of it like this, how will you as a user know when you visit github that another WoSign hasn't happened? ...or rather - how would a LAYPERSON know they are secure? Github might detect it - but how do ordinary USERS?

Yes, we're talking past each other. My argument is that CAA would prevent certain mis-issuances, such as the GitHub/WoSign example. The certificate would not have been issued in my example. It is not a mechanism against a fully-compromised CA, as I have said before. That's what HPKP is for, as I have already said before.

> that (wrongly) assumes ALL CA's trusted by EVERY browser will perform CAA check before issuing a cert for github.

You're wrongly assuming that ALL CAs have the same domain validation vulnerability that WoSign had. I'm saying: Given the WoSign vulnerability, CAA would have probably prevented the mis-issuance.

Also: see previous comments regarding the effectiveness of CAA prior to it being mandatory.

> The problem is not so much "fully-compromised CA" as it is we're trusting a whole pile of CA's and not all them behave the same.

Which is why I have (for the n'th time) indicated that CAA is only fully effective if all CAs implement it (or in other words: if it becomes mandatory).

> Any attacker that can gain access to web site admin credentials can also get the DNS credentials.

What? RCE on the web server is not the same thing as full access to the DNS. You're not making any sense.

> You're using defense-in-depth again. Replace "-in-depth" with "-added-complexity".

Again, you fail to demonstrate how this complexity does more harm than good. You can make this argument for any change.

Re:Is HTTP Public Key Pinning Dead? by jb613 on Tuesday September 06, 2016 @05:11PM (#12439325)
> Still doesn't make any sense to me. Whether a CA performs a DNS query in order to do domain validation via email, http or to check a CAA record doesn't matter.

Brush up on CA cert issuance. You seem to be assuming that all CA's perform similar levels of due diligence before issuing certs. They don't, they differ widely. Some go much further than simply DNS validation.

> I'm suggesting making CAA mandatory.

For practical reasons, I am skeptical this will happen. Too many paying entities. In the spec/contract, the MUSTs will be lowered to SHOULDs.

> The certificate would not have been issued in my example.

IF the CA checked the CAA...

> CAA would have probably prevented the mis-issuance.

Exactly - "probably".

> CAA is only fully effective if all CAs implement it (or in other words: if it becomes mandatory).

1) it won't happen. some CA's may/already-have implemented it but how is the browser/user to know which have and which haven't? 2) unclear that it is even fully effective

> Again, you fail to demonstrate how this complexity does more harm than good. art. Introducing a new mechanism requires demonstration that the added complexity is worth the effort. And in this case, it is clear that unless everyone implements it, there is no added benefit. Added cost without benefit is a bad start

Re:Is HTTP Public Key Pinning Dead? by pfg on Wednesday September 07, 2016 @01:35AM (#12441143)
> They don't, they differ widely. Some go much further than simply DNS validation.

You're missing my point. You're acting like adding yet another DNS query equals some massive increase in complexity. However, all certificate issuance requires DNS queries. I'm quite aware of other validation steps (like for OV and EV). These steps are performed on top of the domain validation steps I mentioned. I have never argued that those are the only steps.

Re:Is HTTP Public Key Pinning Dead? by Strilanc on Tuesday September 06, 2016 @09:08AM (#12435685)
Server software might be able to help with some of the foot-gunning mentioned in Scott Helme's writeup. For example, web servers that were just configured to do cert pinning for the first time could default to giving a 5-hour grace period where the max-age was clamped below a day, followed by a 2-day grace period where the max-age was clamped below a week. Problems are likely to show up immediately, so starting with a low max-age makes sense.

This wouldn't help with malicious scenarios though; attackers would just disable the grace period.

Re:Is HTTP Public Key Pinning Dead? by zrm on Tuesday September 06, 2016 @11:17AM (#12436593)
You could do that on the client side too though. Have the clients refuse to long-term pin anything until they've seen it multiple times over multiple days.
Re:Is HTTP Public Key Pinning Dead? by Strilanc on Wednesday September 07, 2016 @02:02PM (#12445581)
Doing it client-side defeats the purpose of certificate pinning. Three years after you rolled out pinning, new users would still be using max ages that were clamped too low.
Re:Is HTTP Public Key Pinning Dead? by nathanaldensr on Tuesday September 06, 2016 @08:21AM (#12435396)
Just curious: What did you mean by you "did a show of hands?" Do you mean that's the number of people that had heard of or understood CAA?
Re:Is HTTP Public Key Pinning Dead? by eganist on Tuesday September 06, 2016 @08:31AM (#12435451)
Literally:

1) raise your hand if you've heard of CSP (more than half the room at both talks)

2) raise your hand if you've heard of HPKP (half at def con. Probably a third at black hat)

3) raise your hand if you've heard of CAA (I gave this one something like 15 seconds for people to think. The number of hands v. attendees was a rounding error.)

Re:Is HTTP Public Key Pinning Dead? by JoshTriplett on Tuesday September 06, 2016 @08:51AM (#12435558)
Next time you're testing that, you might also include Certificate Transparency.

Mandatory CT seems like it solves the same problems HPKP does, without the risk. Now if we can just finish getting to mandatory CT...

Re:Is HTTP Public Key Pinning Dead? by pfg on Tuesday September 06, 2016 @09:06AM (#12435668)
Certificate Transparency (especially if mandatory) guarantees detection after the fact (if we assume the domain owner is properly monitoring known logs). It also adds a certain chilling effect that will probably cause attackers to look for other methods of compromising the target rather than risk sacrificing one of the CAs under the attacker's control (which, arguably, is what intelligence agencies are already doing as many high-profile sites have deployed HPKP).

It's not a real-time protection mechanism for targeted attacks and such - the mis-issuance event won't be noticed immediately, and revocation is both slow and, well, broken in general.

The existence of HPKP is still a good thing for the Gmail's of this world, but I agree that mandatory CT is probably good enough for 99 out of 100 sites out there.

Re:Is HTTP Public Key Pinning Dead? by JoshTriplett on Tuesday September 06, 2016 @09:51AM (#12435964)
I can imagine extensions to CT that would make it better at combating real-time attacks before they can affect anyone.

For instance, suppose a new certificate didn't become valid for a certain amount of time after the publication of the corresponding CT log, to give the domain owner (or rather, automated processing working on their behalf) time to shoot it down somehow? You could choose how much of a delay you wanted, trading off between security and potential downtime (if you lose all access to your existing certificates, can't revoke them, and have to issue entirely new ones).

If you use any non-automated CA, you should know when renewing your certificates, so you could open up a window to renew; in the absence of that, a fully automated process could notice a new certificate for your domain and automatically shoot it down.

If you use an automated CA, you (or the automated process) should similarly know the window of time you will renew in, though in that case someone could learn that and take advantage of that window. On the other hand, given sufficient confidence in the automated CA you could use CAA with it.

Re:Is HTTP Public Key Pinning Dead? by agwa on Tuesday September 06, 2016 @12:13PM (#12437043)
> such as more diligence by free certificate authorities -- the crux of the talk was rekeying at up to 20x per week enabling the attack

You are wrong if you think this is the crux of the attack. Let's Encrypt is unique in rate limiting issuance to 20 a week. Other CAs will happily give you unlimited reissues once you pay a small amount for the initial cert (e.g. $15), and they have an API for it too!

Re:Is HTTP Public Key Pinning Dead? by eganist on Tuesday September 06, 2016 @01:25PM (#12437636)
Well aware, but not having to pay for it makes it more widely accessible. Any nation-state attacker or attacker with the will to also compromise an identity or two will have no problems either compromising a CA or swiping an owned credit card respectively, but will either of these actors really go out of their way to ransom a site for up to a five figure sum? The former has no incentive to make money, and the latter makes money through identity theft as-is.

For what it's worth, we're aware of rapid rekeying with other CAs because Cyph uses rapid key rotation with DigiCert right now and has since before Let's Encrypt entered open beta.

Re:Is HTTP Public Key Pinning Dead? by agwa on Tuesday September 06, 2016 @02:12PM (#12438006)
> Well aware, but not having to pay for it makes it more widely accessible. Any nation-state attacker or attacker with the will to also compromise an identity or two will have no problems either compromising a CA or swiping an owned credit card respectively, but will either of these actors really go out of their way to ransom a site for up to a five figure sum? The former has no incentive to make money, and the latter makes money through identity theft as-is.

These CAs have resellers that sell certs anonymously, accepting Bitcoin as payment. If the upside is a five-figure sum, $15 is nothing.

> For what it's worth, we're aware of rapid rekeying with other CAs because Cyph uses rapid key rotation with DigiCert right now and has since before Let's Encrypt entered open beta.

Odd then that you mentioned Let's Encrypt 5 times in your Chrome security report (and reported this to the Let's Encrypt security team, as if it's their problem), but didn't mention DigiCert once: https://bugs.chromium.org/p/chromium/issues/detail?id=620776

Re:Is HTTP Public Key Pinning Dead? by eganist on Tuesday September 06, 2016 @10:37PM (#12440668)
> Odd then that you mentioned Let's Encrypt 5 times in your Chrome security report (and reported this to the Let's Encrypt security team, as if it's their problem), but didn't mention DigiCert once: https://bugs.chromium.org/p/chromium/issues/detail?id=620776

It's not odd when you consider the point I'm re-quoting below (which isn't really negated by Bitcoin reselling since Bitcoin anonymization is non-trivial and since it still requires payment):

> Well aware, but not having to pay for it makes it more widely accessible.

As you noted, the bug report is readily accessible for anyone to view, and it entirely supports the quoted assertion above. In the end, we'd rather draw attention to the free CAs and HPKP-supporting user agents first and foremost in hopes of locking down the problem where it's most conveniently exploited because it is their problem, at least in our eyes. We can tackle the resellers later, and the entire problem of the broken CA model thereafter.

Hopefully that satisfies you.

Edit: shot you an email. Something from your tone strikes me as this being a personal matter for you. Our goal here isn't to antagonize anyone, just to make sure we've done our due diligence in disclosing the issue to those on the path of least resistance for attackers.