News for hackers. Stuff that matters.
Re:Encrypted traffic interception on Hetzner and Linode targ... by rodlette on Friday October 20, 2023 @12:34PM (#37958299)
> All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names

+1. crt.sh has RSS feeds for this.

> Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods

Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?

> Monitor SSL/TLS certificate changes on all your services using external service

+1. High-risk targets should be aware of what certs are valid at any time, and be checking for those.

> Monitor MAC address of default gateway for changes

A more sophisticated attack could preserve the MAC address.

> "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate. Both the client and the server must support SCRAM PLUS authentication mechanisms for this to work. Unfortunately this was not active on jabber.ru at the time of the attack.

TIL.

I'd also add to these, consider an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, or something like Noise Pipes http://www.noiseprotocol.org/noise.html#noise-pipes

Re:Encrypted traffic interception on Hetzner and Linode targ... by hlandau on Friday October 20, 2023 @01:16PM (#37958831)
>Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?

Author of ACME-CAA (RFC 8657) here. ACME-CAA can mitigate this because you can put a unique identifier for your ACME account in the CAA record, so it is not possible for an attacker to do this unless they can get your ACME account private key (or coerce the ACME service). This assumes you have DNSSEC-secured nameservers, of course, otherwise DNS requests can potentially also be intercepted when queried by the CA.

See RFC 8657 for a full list of security caveats. The RFC is designed to be more readable than most.

Blogpost by me with more background: https://www.devever.net/~hl/acme-caa-live

Re:Encrypted traffic interception on Hetzner and Linode targ... by aaomidi on Friday October 20, 2023 @02:23PM (#37959676)
Yep! This is one way of preventing this type of attack.
Re:Encrypted traffic interception on Hetzner and Linode targ... by crosser on Friday October 20, 2023 @02:35PM (#37959798)
Could you please elaborate the reasons to implement this outside of DANE (RFC-7671) framework?
Re:Encrypted traffic interception on Hetzner and Linode targ... by tialaramex on Friday October 20, 2023 @02:57PM (#37960043)
DANE assumes we can successfully deploy this to the entire Internet. It is unclear that's ever possible, and it's certainly not possible today. Lots of things would be great if you can deploy them, for example you wouldn't build QUIC on top of UDP since you can "just" deploy a new transport protocol - except nope, for the foreseeable future that's undeployable.

A public CA generally has a more sophisticated relationship with their network transit provider or (hopefully) providers and can get DNSSEC actually working as intended for them.

So this means mything.example's DNS services and some public CA both need working DNSSEC, but the visitors to mything.example, such as your mom's sister or some guy who just go into mything but isn't entirely clear whether Apple make Windows - do not need DNSSEc, for them everything works exactly as before, yet the certificate acquisition step is protected from third parties.

Would that help? It depends.

Re:Encrypted traffic interception on Hetzner and Linode targ... by fanf2 on Friday October 20, 2023 @03:01PM (#37960086)
CAA is about preventing certificate mis-issuance, which is what happened in this attack. DNSSEC and CAA could have prevented this attack from being performed the way it was, by thwarting the MITM on ACME.

DANE is about changing the way certificates are authenticated. DANE makes it possible to authenticate certificates without getting them issued by a well-known CA. So CAA records are not particularly relevant to DANE. You can use DANE with certificates issued by a CA, which gives you two ways to authenticate the certificate; in this situation CAA secures one path and DANE the other.

I am one of the co-authors of the DANE SRV RFC https://www.rfc-editor.org/rfc/rfc7673 which is what XMPP would use. I don’t follow XMPP development so I don’t know if it has been deployed. I would like it if DANE were more widely used, but it’s not pertinent to this attack.

Re:Encrypted traffic interception on Hetzner and Linode targ... by crosser on Friday October 20, 2023 @03:50PM (#37960706)
Why not? You could use "certificate usage" value 1 and (if the implementation does not neglect it) immediately notice that validation by CA disagrees with validation by DNS. That should be good enough, no?
Re:Encrypted traffic interception on Hetzner and Linode targ... by hlandau on Friday October 20, 2023 @04:13PM (#37960961)
Yeah. I used to be 100% in on DANE and against CAs. I'm still 100% for DANE but I now think DANE using existing CAs is the better option in many cases because it means things get CT logged. We don't have a DNSSEC transparency situation right now. OTOH there is one undersung issue with CAs, which is that Let's Encrypt isn't as universally available as people think (see the US embargo list) and that does potentially make access to the internet harder for some.

There are some use cases where DANE is actually winning real victories and is actually more viable than the existing CA infrastructure - site-to-site SMTP, for example.

Re:Encrypted traffic interception on Hetzner and Linode targ... by fanf2 on Friday October 20, 2023 @04:58PM (#37961434)
Yeah, Viktor Dukhovni has been impressively energetic and persistent at improving the security of email.
Re:Encrypted traffic interception on Hetzner and Linode targ... by KirillPanov on Saturday October 21, 2023 @04:52AM (#37965184)
I feel like packet size was and continues to be a major obstacle for DNSSEC. Do you know why the DNSSEC/DANE world hasn't simply acknowledged this and switched to requiring ECC?

It is trivial to fit several compressed curve points (i.e. signatures) in a single packet, whereas you can't even fit two RSA signatures in a minimum-safe-to-assume DNS UDP reply packet after accounting for padding and ASN.1 overhead.

I get the feeling that there is some faction that really hates UDP and they sort of hijacked the DNSSEC situation to use as a lever to force people to allow DNS-over-TCP.

That seems to be backfiring, however, and DNSSEC has wound up taking a bullet for the UDP-haters.

Many very-large networks simply can't afford for their DNS traffic to be exposed to TCP's intrusion-detection malleability and slowloris (resource exhaustion) attacks. These networks appear to be simply ignoring the "thou must TCP thine DNS" edict. DNSSEC is not a good enough carrot for them. I think ditching RSA would have been a more pragmatic choice than ditching UDP or skipping DNSSEC.

Re:Encrypted traffic interception on Hetzner and Linode targ... by fanf2 on Saturday October 21, 2023 @12:09PM (#37968011)
Dunno why there are so many foot-draggers failing to deploy better DNSSEC algorithms. I’m grumpy about SHA-1 in particular https://datatracker.ietf.org/doc/html/draft-fanf-dnsop-sha-l...

When I query vjhv.verisign.com I get a response containing four 2048 bit RSA-SHA-2 signatures in 1049 bytes which is well within the EDNS MTU for unfragmented UDP, so I’m not convinced the problem is as bad as you paint it. There have been problems with EDNS trying to use fragmented UDP, but that has been reduced a lot by newer software being more cautious about message size limits for DNS over UDP.

DNS needs TCP even in the absence of DNSSEC, because there are queries you cannot resolve without it. Some operators might convince themselves they can get away without it, but they will probably suffer subtle breakage.

Re:Encrypted traffic interception on Hetzner and Linode targ... by KirillPanov on Saturday October 21, 2023 @07:56PM (#37971613)
> four 2048 bit RSA-SHA-2 signatures in 1049 bytes which is well within the EDNS MTU for unfragmented UDP

I was referring to the non-EDNS 512-byte limit.

Yes, you get ~2.5 times more with EDNS. Still, four records is not a lot.

> DNS needs TCP even in the absence of DNSSEC, because there are queries you cannot resolve without it.

Theoretically? Perhaps. Some would argue that connectionless DNS is valuable enough that people should not create those resource records. Before DNSSEC that was a working consensus. And with ECC it could be once again.

Re:Encrypted traffic interception on Hetzner and Linode targ... by tptacek on Sunday October 22, 2023 @05:57PM (#37979460)
That's the opposite of the direction Internet cryptography is going, given hybrid PQC and classical systems.
Re:Encrypted traffic interception on Hetzner and Linode targ... by KirillPanov on Tuesday October 24, 2023 @04:28PM (#38005017)
The bloaty key/signature size is only a problem with the PQ encryption systems.

For signing only there are much more efficient PQ cryptosystems, with signatures around the same size as ECC. If DNSSEC ever adopts PQC it will be one of those systems.

Here are two of the earliest, and easiest to understand. There are much better ones now.

https://en.wikipedia.org/wiki/Lamport_signature#Short_keys_a...

https://en.wikipedia.org/wiki/Merkle_signature_scheme

Re:Encrypted traffic interception on Hetzner and Linode targ... by tptacek on Sunday October 22, 2023 @05:58PM (#37979467)
Mostly, it's because very few serious engineering organizations deploy DNSSEC at all, so the best practices and tooling support aren't there.
Re:Encrypted traffic interception on Hetzner and Linode targ... by singpolyma3 on Saturday October 21, 2023 @09:55AM (#37966813)
Unfortunately the DANE SRV RFC is kind-of mismatched with how SRV and TLS work in practise. It requires the server to serve a certificate matching its own hostname (the hostname of the SRV target) rather than a certificate matching the expected host (the hostname that the SRV record was on). This is fine and secure if you use only DANE but if you want to use DANE with CA-issued certs it makes it somewhere between hard and impossible.
Re:Encrypted traffic interception on Hetzner and Linode targ... by fanf2 on Saturday October 21, 2023 @11:47AM (#37967764)
Note the owner of a SRV record is a service name not a host name.

There are a few reasons for this oddity: partly so it matches with DANE for MX records, partly to support large scale virtual hosting without reissuing certificates.

You should be able get a cert with subject names covering the server host name(s) and the service name(s).

Re:Encrypted traffic interception on Hetzner and Linode targ... by hlandau on Friday October 20, 2023 @04:34PM (#37961184)
Small update: I've just written a blogpost with my thoughts on the incident. https://www.devever.net/~hl/xmpp-incident

I usually brood on blogposts for days before publishing them since I care a lot about getting things accurate, but this is a bit of the moment, so here goes. Always happy to get feedback by email or IRC: https://www.devever.net/~hl/contact

Re:Encrypted traffic interception on Hetzner and Linode targ... by ThePowerOfFuet on Friday October 20, 2023 @06:02PM (#37962010)
Is it just me, or would a recurring RIPE Atlas measurement be a great way to detect fuckery like this?

https://atlas.ripe.net/

Re:Encrypted traffic interception on Hetzner and Linode targ... by bananapub on Friday October 20, 2023 @07:30PM (#37962651)
not at all, since there were no Internet-visible routing changes.

atlas would be good for detecting the time Pakistan announced 0/0 or whatever.

Re:Encrypted traffic interception on Hetzner and Linode targ... by karmahunter1234 on Saturday October 21, 2023 @12:02AM (#37963970)
I think atlas allows tls/ssl probes, so that could still be used to track unexpected changes in certificates?
Re:Encrypted traffic interception on Hetzner and Linode targ... by 3np on Friday October 20, 2023 @06:35PM (#37962243)
[delayed]
Re:Encrypted traffic interception on Hetzner and Linode targ... by hlandau on Friday October 20, 2023 @07:22PM (#37962595)
So I assume DLT is a generic/more general term for blockchain technology.

There's this thing called Zooko's triangle (https://en.wikipedia.org/wiki/Zooko%27s_triangle), basically the premise that an identifier can only have two of the following properties: secure, decentralized, human-readable.

But a blockchain can actually be used to square this triangle and get all three. The original example of this is the Namecoin project, a fork of Bitcoin using a lightly modified Bitcoin codebase which can be used as a key-value store.

The idea is that a blockchain can be used to create a decentralized database mapping keys (domain names) to values (which can be things like IP addresses, but also PKI trust anchors, similar to DANE). Thus this can eliminate the need for a CA. You can also use it to map human-readable names to .onion addresses. There is also root-of-trust transparency built in since the contents of the database is public and any changes are also public. The right to change a key-value entry belongs to the public key of the person who registered that key. Nobody can override this. Namecoin domain names use the .bit (unofficial) TLD.

It's a really neat technology and also makes things like censorship via domain suspensions, your registrar getting hacked, etc. infeasible.

Full disclosure, I previously worked on the Namecoin project, I authored the DNS resolution daemon and the technical specification for the format of DNS data in the KV store. Unfortunately public interest in the project has waned, and deployment is the real issue - you have to have the client software to be able to resolve these domain names. There is at least some conceivable possibility Tor might ship it in their bundle in the future though, so people could type example.bit in Tor Browser to get to a hidden service. The project remains active though.

Re:Encrypted traffic interception on Hetzner and Linode targ... by Groxx on Friday October 20, 2023 @07:13PM (#37962527)
Interesting and I can see some of the benefits, particularly in preventing DNS controllers from going to other CAs and making new keys, but... this all seems like a weird run-around for the MITM part.

Couldn't you exchange public keys with LetsEncrypt (in the web UI) and encrypt the response so you can't be MITMed? Why is http even an issue?

Re:Encrypted traffic interception on Hetzner and Linode targ... by KairuByte on Friday October 20, 2023 @10:17PM (#37963505)
Because the attacker can simply request new keys. There’s nothing stopping them from going “hey LE, I need a new key! This is my domain, here is the challenge, give me my cert!” And LE will oblige, because as far as it can tell, they are you.

Edit: To be clear, this is a problem with a solution. But you asked why simply throwing a LE cert into the mix wouldn’t prevent the issue.

Re:Encrypted traffic interception on Hetzner and Linode targ... by Groxx on Friday October 20, 2023 @10:28PM (#37963552)
Ownership is already handled by checking DNS (and this thread covers a way to make that even more secure, which LE supports), and as far as I can tell neither has nothing to do with preventing MITM between LE and your servers.

And no, I don't mean throwing LE certs around to prevent MITM - this whole article is about the difficulties before having an LE cert, so that's necessarily excluded.

I'm wondering "why not client certificates". They're a well established way to stop MITM, seems like a simple choice for the ownership validation step.

Re:Encrypted traffic interception on Hetzner and Linode targ... by salawat on Saturday October 21, 2023 @02:33PM (#37969336)
Have you ever tried to explain key distribution/management to a normal person?

You should try it sometime.

Re:Encrypted traffic interception on Hetzner and Linode targ... by Groxx on Monday October 23, 2023 @06:40PM (#37992588)
[delayed]
Re:Encrypted traffic interception on Hetzner and Linode targ... by 0xbadcafebee on Friday October 20, 2023 @02:54PM (#37960004)
> All issued SSL/TLS certificates are subject to certificate transparency.

Until the nation state tells a CA within their borders to issue a cert without publishing the CT logs, along with a gag order. If they don't comply they go to jail. You think Billy Bob Sysadmin wants to go to jail to protect a russian jabber server?

Re:Encrypted traffic interception on Hetzner and Linode targ... by bart__ on Friday October 20, 2023 @03:43PM (#37960616)
Browsers will check if a certificate is in the transparency log, and alert the user if it isn't if I am not mistaken.
Re:Encrypted traffic interception on Hetzner and Linode targ... by crtasm on Friday October 20, 2023 @05:03PM (#37961491)
But XMPP clients do not, as far as I'm aware? and browsers aren't connecting to XMPP server ports.
Re:Encrypted traffic interception on Hetzner and Linode targ... by est31 on Friday October 20, 2023 @05:38PM (#37961852)
Yeah usually the TLS libraries used by XMPP clients don't check SCTs. Not even all browsers do it. Chrome does it, Firefox does not for example.
Re:Encrypted traffic interception on Hetzner and Linode targ... by privacyking on Saturday October 21, 2023 @12:10AM (#37964009)
Source?
Re:Encrypted traffic interception on Hetzner and Linode targ... by bart__ on Saturday October 21, 2023 @06:08AM (#37965508)
https://developer.mozilla.org/en-US/docs/Web/Security/Certif...

As the other commenter already pointed out, Firefox does not require this. Safari and Chrome do. This indeed is not directly applicable to this situation, since XMPP don't involve browsers. But for websites, the parents attack scenario is not applicable.

Re:Encrypted traffic interception on Hetzner and Linode targ... by tgsovlerkhgsel on Friday October 20, 2023 @05:05PM (#37961511)
The malicious certificate either won't have a SCT (which in itself would be highly suspicious and cause some clients to reject the certificate directly), or it will have one but it won't be recorded in the log.

In the latter case, the malicious certificate + any CT log issued after that certificate should have been included are evidence of the attack that is easily verifiable and would likely cause browsers to drop the (edit) CT log unless provided with a very plausible excuse that isn't "we complied with a court/gag order".

Also, this would require the police etc. to compel 2-3 different entities: The CA and two log operators (I think one of them could be the CA itself).

Using different logs than normal would stand out, so it'd typically be two specific log operators that would have to be compelled to hide this. The US may get lucky and have jurisdiction over all of them, other countries are much less likely to.

Re:Encrypted traffic interception on Hetzner and Linode targ... by aaomidi on Saturday October 21, 2023 @11:17AM (#37967511)
Even if the US got lucky, those SCTs wouldn’t match the CT chain and would still be discoverable misissuance.
Re:Encrypted traffic interception on Hetzner and Linode targ... by tgsovlerkhgsel on Sunday October 22, 2023 @03:15AM (#37973480)
Yes, assuming someone captured the malicious certificate and checked. Which would have happened in this case now (after someone forgot to renew), but likely not before.

Browsers would happily accept it and IIRC they don't check against the logs.

Re:Encrypted traffic interception on Hetzner and Linode targ... by aaomidi on Sunday October 22, 2023 @10:56AM (#37975892)
Yep - this is all because of the delayed insertion into the CT tree :(
Re:Encrypted traffic interception on Hetzner and Linode targ... by account42 on Monday October 23, 2023 @04:13AM (#37982819)
> and would likely cause browsers to drop the (edit) CT log unless provided with a very plausible excuse that isn't "we complied with a court/gag order"

a) Browsers aren't the only clients that matter. Will any XMPP client even look at the CT log? Probably not.

b) Browsers are not going to drop all US-based certificate authorities. And if they tried they might just get their own national security letter (reminder: all major browsers are US-based).

You can't work around the government using centralized infrastructure.

Re:Encrypted traffic interception on Hetzner and Linode targ... by morpheuskafka on Friday October 20, 2023 @06:07PM (#37962043)
If the certificate isn't logged in CT, it's not supposed to be trusted. That's the whole point. If it is logged, then the site owners would notice (Cloudflare, for example, has a service that emails when certificates are issued for your domains).
Re:Encrypted traffic interception on Hetzner and Linode targ... by schoen on Saturday October 21, 2023 @05:18AM (#37965286)
Just to be clear (with better details already given by other comments), addressing this is specifically part of the threat model for Certificate Transparency. You could argue that CT policies should be strengthened in various ways, but the threat model does intend to address detecting the compelled misissuance scenario.
Re:Encrypted traffic interception on Hetzner and Linode targ... by jojobas on Thursday November 02, 2023 @07:30AM (#38111926)
CA being in one country and hosting in another greatly complicates this. Ideally the two countries shouldn't be part of Five Eyes or some such.
Re:Encrypted traffic interception on Hetzner and Linode targ... by abigail95 on Friday October 20, 2023 @06:06PM (#37962031)
If they have physical access to your server, they'll just go for that.

With mild difficulty you can mirror the disk, net, and RAM of any "commodity" hardware.

Re:Encrypted traffic interception on Hetzner and Linode targ... by codedokode on Friday October 20, 2023 @07:01PM (#37962446)
The main problem is that Let's Encrypt issued fake certificates without strict checking. This means any country can start using this method to generate valid certificates for any site within the country. And if the country can pull BGP attack with traffic redirection then for any website.

Let's Encrypt security is a joke.

UPD: It looks like many other CAs also can be tricked to issue valid certificates to any attacker capable of doing MitM, so in general SSL infrastructure security is a joke which is proven by this incident.

Re:Encrypted traffic interception on Hetzner and Linode targ... by lxgr on Friday October 20, 2023 @07:40PM (#37962704)
How would you implement “strict checking” at scale and for free?
Re:Encrypted traffic interception on Hetzner and Linode targ... by codedokode on Friday October 20, 2023 @07:56PM (#37962794)
If they used DNS validation it would be better because there would be only one point of failure (DNS) rather than many (DNS and every ISP between CA and a site).
Re:Encrypted traffic interception on Hetzner and Linode targ... by Jenda_ on Friday October 20, 2023 @09:07PM (#37963202)
The thread above (https://news.ycombinator.com/item?id=37958831) elaborates on how to force DNS validation and/or how to tie your private key with Let's Encrypt via DNS.
Re:Encrypted traffic interception on Hetzner and Linode targ... by codedokode on Saturday October 21, 2023 @01:14PM (#37968659)
Everyone needs to opt-in to use more secure methods and by default non-secure validation methods, which allow easy issuing fake certificates, are allowed. This is wrong.
Re:Encrypted traffic interception on Hetzner and Linode targ... by Jenda_ on Saturday October 21, 2023 @01:38PM (#37968917)
So, what should they do? No certificate without DNS record? Would this really help the overall state of affairs, or would most sites just not use HTTPS at all because it's "too complicated"?
Re:Encrypted traffic interception on Hetzner and Linode targ... by codedokode on Saturday October 21, 2023 @02:07PM (#37969163)
The purpose of using HTTPS is to make connections more secure. Giving away SSL certificates to anyone does not serve this purpose.
Re:Encrypted traffic interception on Hetzner and Linode targ... by lxgr on Saturday October 21, 2023 @02:43PM (#37969436)
It absolutely serves this purpose in a world in which there unfortunately is no TOFU/unauthenticated encryption for TLS (i.e. ours).

Thanks to widely available HTTPS certificates, "evil hackers stealing your cookies on public Wi-Fi" is not a thing anymore.

We should definitely have a discussion about whether it's made active attacks more feasible, but I think the goal of making passive sniffing less trivial than it was before can be considered achieved.

Re:Encrypted traffic interception on Hetzner and Linode targ... by lxgr on Friday October 20, 2023 @09:28PM (#37963293)
And DNS does not traverse several ISPs?
Re:Encrypted traffic interception on Hetzner and Linode targ... by kakwa_ on Saturday October 21, 2023 @09:21AM (#37966610)
Implement a proper Proof of Possession system (PoP) and make Let's Encrypt opt-in instead of opt-out.

Given that in the end, Let's Encrypt/ACME relies on PoP of a DNS domain, DNSSEC setup on the domain should have been a prerequisite.

In addition, an explicit opt-in should be required, for example, with a requirement for a CAA record pointing to LE.

---

As a side note, this whole situation leaves me with a really weird feeling.

On one end, I will always be grateful for LE enabling tls/https generalization.

On the other, I kind of feel betrayed and/or ashamed that LE/ACME almost willingly introduced protocol flaws weakening the whole CA infrastructure and that we (me included) didn't challenge it more when LE was introduced.