News for hackers. Stuff that matters.
Re:2021.06.08 Certificate Lifetime Incident by jaas on Wednesday June 09, 2021 @02:31PM (#27451379)
Head of Let's Encrypt here.

The question of whether or not revocation should happen has to be asked whenever a certificate compliance issue is being discussed, regardless of how serious the issue is. That is a normal part of the process of evaluating an incident thoroughly.

We do not plan to revoke any certificates as a result of this issue.

An aside - I love how informed many of the commenters here are, thanks to you all for helping to explain what's happening!

Re:2021.06.08 Certificate Lifetime Incident by jiggawatts on Wednesday June 09, 2021 @06:17PM (#27454021)
I love your work!

Any chance you could convince Microsoft to add Let's Encrypt as an "integrated" CA in Azure Key Vault? It's absolutely bonkers how much money I have to pay to get a certificate in 2021 for cloud services! E.g.: the App Service certificates are $70/year each or $300/year for a wildcard certificate. That's nuts. Reference: https://azure.microsoft.com/en-us/pricing/details/app-servic...

I strongly suspect the reason Let's Encrypt isn't adopted more widely in cloud services is because there's no margin on a free service. This is why Microsoft, AWS, and GCP all carefully pretend that there are no free options, and make sure that it's a difficult uphill battle to use Let's Encrypt.

E.g.: https://docs.microsoft.com/en-us/azure/key-vault/certificate...

Notice how the document title is literally "Integrating Key Vault with DigiCert certificate authority". Not "Certificate Authority", it's the "DigiCert certificate authority". Apparently, HTTPS is now DigiCert's protocol, they're the gatekeepers, and you have to pay them money to use it.

It boils my blood that 1KB files of random numbers still cost money, and the trolls under the bridge are still taxing everyone for what is now essentially mandatory for all web sites.

If anyone here has a significant account with Azure, please apply some pressure to your Microsoft account manager next time you have coffee with them. This rent seeking for what should be free for everyone has to stop.

Re:2021.06.08 Certificate Lifetime Incident by Thorrez on Wednesday June 09, 2021 @06:50PM (#27454347)
>This is why Microsoft, AWS, and GCP all carefully pretend that there are no free options, and make sure that it's a difficult uphill battle to use Let's Encrypt.

Maybe I'm misunderstanding, but GCP HTTPS and SSL load balancers give you certificates for free. They support Google's own CA (pki.goog) as well as Let's Encrypt. Full disclosure I work in GCP, but not on this.

https://cloud.google.com/load-balancing/docs/ssl-certificate...

Re:2021.06.08 Certificate Lifetime Incident by pmoleri on Wednesday June 09, 2021 @08:30PM (#27455269)
AWS load balancers also give you managed certificates for free.
Re:2021.06.08 Certificate Lifetime Incident by irgeek on Wednesday June 09, 2021 @06:36PM (#27454200)
Certificates issued from AWS Certificate Manager are — and, I believe, always have been — free. AWS is definitely not rent-seeking on certificates.
Re:2021.06.08 Certificate Lifetime Incident by NovemberWhiskey on Wednesday June 09, 2021 @06:47PM (#27454318)
Certificates from AWS Certificate Manager are part of the roach motel though - AWS orchestrates the issuance and holds the private keys so you can only use them via AWS services.
Re:2021.06.08 Certificate Lifetime Incident by Deathmax on Wednesday June 09, 2021 @06:57PM (#27454424)
But that's what the grandparent was complaining about though? If you're not using the provider's managed services, then nothing is stopping you from running your own ACME client to provision certificates without paying the cloud provider money for certs.
Re:2021.06.08 Certificate Lifetime Incident by jiggawatts on Wednesday June 09, 2021 @07:29PM (#27454748)
"AWS Certificate Manager supports a growing number of AWS services. You cannot install your ACM certificate or your private ACM Private CA certificate directly on your AWS based website or application."

Free certificates you can't use on EC2 virtual machines are basically worthless, at least for me.

Stop internalising your 1990s architecture limitations! You shouldn't need to pay for a Layer 7 load balancer for an application that doesn't need it. A 1-core web server VM can easily put out 1-2 Gbps of HTTPS traffic. You don't need SSL offload. A crypto accelerator card is not required. You don't need an appliance to do HTTPS. You can have end-to-end HTTPS without additional infrastructure. Both Windows and Linux can do TLS out-of-the-box. You don't need a vendor to give you special permission to have security. There is no need to pay GoDaddy or DigiCert for a certificate.

The vendors are pulling the wool over your eyes, convincing you that your out-of-date thinking is good and proper, and then charging you for the privilege of having the bare minimum security that should be free as standard.

Re:2021.06.08 Certificate Lifetime Incident by efitz on Wednesday June 09, 2021 @08:05PM (#27455063)
AWS ACM has always been free. ACM has never allowed you to export private keys. However ACM does have a way[1] for you to use ACM keys with EC2 instances: Nitro Enclaves. Nitro Enclaves carve off a little piece of your EC2 instance (memory + VCPU) into an isolated VM that feels a little bit like an HSM or a secure enclave.

[1] https://docs.aws.amazon.com/enclaves/latest/user/nitro-encla...

Re:2021.06.08 Certificate Lifetime Incident by marcinzm on Wednesday June 09, 2021 @08:37PM (#27455330)
If it's your own EC2 instance and you're not using a load balancer then why can't you just use Let's Encrypt?
Re:2021.06.08 Certificate Lifetime Incident by fomine3 on Wednesday June 09, 2021 @10:29PM (#27456047)
Just use certbot or alternatives, what's the problem?
Re:2021.06.08 Certificate Lifetime Incident by brazzledazzle on Wednesday June 09, 2021 @09:40PM (#27455763)
Holy shit, Azure really charges that much? Does the tls certificate come with some other service or unique feature bundled as part of the cost? AWS basically charges you for the API calls, cents really, and I’m fairly certain GCP is at least as inexpensive if not free. Can you at least export it and take it with you? That’s really the trade off with AWS, private keys aren’t exportable. But it’s so cheap that it’s a non issue. If I want one I can manage myself I can pay or use letsencrypt.
Re:2021.06.08 Certificate Lifetime Incident by mattmanser on Wednesday June 09, 2021 @06:35PM (#27454183)
AFAIK Azure do free ones now, I'm certainly using some free certs on Azure.
Re:2021.06.08 Certificate Lifetime Incident by jiggawatts on Wednesday June 09, 2021 @07:18PM (#27454643)
They're not really free: "This feature is available for customers on an App Service Plan of Basic and above (free and shared tiers are not supported)." They're GoDaddy certificates, and their price is charged back through the App Service pricing.

Similarly, Azure Front Door also has "free" certificates, but they just integrate it into the relatively high cost of the service.

If you want certificates for some other unrelated IaaS or PaaS service... Microsoft says no. They want their margin.

Back to GoDaddy: their attitude is very 1990s, so they sometimes use manual approval for certificates. This makes ARM Templates that normally take minutes to deploy just hang and take hours, or even fail.

Worse, they don't use the DNS address in your request for validation of domain ownership. Instead, they determine the "TLD+1" with some heuristics. Unfortunately, there is no such concept in the Domain Name System itself, so this is unreliable at best. This approach is broken by design and cannot be made to work for many domains. For example, in Australia, App Service Certificates cannot ever be used for subdomains of act.gov.au, nsw.gov.au, and nt.gov.au!

PS: For people who are unaware, the concept of the TLD is at best a fuzzy one, and is decided by the informally maintained Public Suffix list, which is currently managed by Mozilla. It's not an RFC, it's not a standard, and isn't suitable for certificate validation. See: https://publicsuffix.org/

This is one of the key philosophical differences between Let's Encrypt and GoDaddy. When issuing automated, free certificates, manual labour for validation is not a viable approach and hence Let's Encrypt eliminated all such sources of informal, error-prone, manually verified sources. GoDaddy hasn't changed their validation approach in decades, because for $70/year, this kind of inefficiency is acceptable.

To put things in perspective: GoDaddy has a support phone number. For certificates! They're literally 1KB files with two numbers and some text in them. Why do they need support!?

Re:2021.06.08 Certificate Lifetime Incident by DaiPlusPlus on Wednesday June 09, 2021 @09:06PM (#27455544)
> Why do they need support!?

People manually installing certificates who are unfamiliar with the process. Which, to be fair, is a complex and very technical process if it’s some self-hosted or shared-hosting shell account running an outdated version of cPanel.

Re:2021.06.08 Certificate Lifetime Incident by brazzledazzle on Wednesday June 09, 2021 @09:44PM (#27455789)
I’m kind of shocked that Microsoft has to partner with godaddy and didn’t already have their own CA long before Azure was even a twinkle in some executive’s eye.
Re:2021.06.08 Certificate Lifetime Incident by akersten on Wednesday June 09, 2021 @04:33PM (#27452843)
Thanks for Let's Encrypt, it's truly an admirable service.

I'm curious regarding "certificate compliance" - I thought the 90-day expiration was merely a Let's Encrypt policy to encourage good automation. Is this just a matter of holding yourselves to high standards, or is there a greater authority to which LE promised 90-days-exactly?

Re:2021.06.08 Certificate Lifetime Incident by jaas on Wednesday June 09, 2021 @05:06PM (#27453227)
90 days is our choice, 90 days and one second validity isn’t necessarily an issue.

The issue is that we said in our CPS that the lifetime was exactly 90 days, and then we did something different, even if just by one second. The problem here is behavioral consistency with our own published policies.

We now just say “less than 100 days.” An unfortunate side effect of being more specific and informative in these documents (e.g. saying our certs have 90 days validity) is that it ups our chances of noncompliance if we are off on something by a bit, even if it is a meaningless bit. There is an incentive to not be so specific so as to avoid situations like this.

Re:2021.06.08 Certificate Lifetime Incident by sigzero on Wednesday June 09, 2021 @02:45PM (#27451539)
That you for the clarification and the letsencrypt service.
Re:2021.06.08 Certificate Lifetime Incident by zaxomi on Wednesday June 09, 2021 @03:27PM (#27452029)
Hi! Is your time servers synchronized to GMT (mean solar time at the Royal Observatory in Greenwich) or UTC (Coordinated Universal Time)?

The RFC requires GMT, but most time servers synchronizes to UTC. It could be a difference of up to 0.9 seconds between GMT and UTC.

Re:2021.06.08 Certificate Lifetime Incident by wahern on Wednesday June 09, 2021 @03:57PM (#27452357)
> It could be a difference of up to 0.9 seconds between GMT and UTC.

Not in this context. In the context of ASN.1, NTP, POSIX, etc, GMT is a timezone equivalent to UTC+00:00. UT1 and DUT1 don't figure into it.

Re:2021.06.08 Certificate Lifetime Incident by fanf2 on Wednesday June 09, 2021 @08:10PM (#27455105)
GMT was abolished at the end of 1971, when the Greenwich observatory started distributing UTC as its official timescale. At that time the observatory was based in Herstmonceux, so GMT was calculated from a model rather than being based on observations made from the transit instruments in Greenwich.
Re:2021.06.08 Certificate Lifetime Incident by u8mybrownies on Wednesday June 09, 2021 @03:19PM (#27451937)
Thank you for let’s encrypt. Very grateful for your role in society.
Re:2021.06.08 Certificate Lifetime Incident by failwhaleshark on Wednesday June 09, 2021 @09:54PM (#27455865)
Won't it basically fix itself in 90 days and 1 second after all the certs are rolled anyhow now that it's on the radar?