News for hackers. Stuff that matters.
Re:All extensions disabled due to expiration of intermediate... by revvx on Friday May 03, 2019 @10:36PM (#19823994)
> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expiry disasters?

LetsEncrypt renewal is supposed to be automated. [1]

I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.

[1] https://letsencrypt.org/about/

Re:All extensions disabled due to expiration of intermediate... by tedunangst on Friday May 03, 2019 @11:43PM (#19824303)
I have no idea why you'd deliberately wait the full 90 days to do a manual renew. For reasons, I renew manually, but every 60 days or so. Nowhere close to the deadline.
Re:All extensions disabled due to expiration of intermediate... by revvx on Friday May 03, 2019 @11:48PM (#19824319)
Exactly.

Their FAQ [1] recommends exactly that: renewing every 60 days.

[1] https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-l...

Re:All extensions disabled due to expiration of intermediate... by n42 on Saturday May 04, 2019 @12:17AM (#19824403)
Just curious, are you talking about Webflow? Because I had to hunt down and make sure our Let's Encrypt auto renewal was working until I realized the certificate was served by them. They wait until the last 12 hours to renew the certificate. I have no idea what type of rationalization would lead to that decision.
Re:All extensions disabled due to expiration of intermediate... by tass on Saturday May 04, 2019 @01:59AM (#19824727)
90 days is 4 times a year. 60 is 6 times, 50% more expensive when you’re paying someone to perform the task.
Re:All extensions disabled due to expiration of intermediate... by n42 on Saturday May 04, 2019 @02:17AM (#19824768)
I had the same thought, but I still find that absurd. Say they host 500,000 websites with HTTPS. 1,000,000 renewals they save spread across the year, roughly 2 renewals a minute. That is pennies. A t2.medium could handle that type of load increase
Re:All extensions disabled due to expiration of intermediate... by albru123 on Saturday May 04, 2019 @02:32AM (#19824806)
A bit OT, but what's up with this usage of Amazon EC2 tiers as a unit of computational power?
Re:All extensions disabled due to expiration of intermediate... by rickycook on Saturday May 04, 2019 @03:58AM (#19825079)
i think it’s a combined “fixed cost” rather than just computational power... like you could do it with x, thus it should cost at most y

similar to saying that you could do it with a raspberry pi

Re:All extensions disabled due to expiration of intermediate... by rocqua on Saturday May 04, 2019 @03:35AM (#19825023)
It is a clearly priced unit of computational power maybe?
Re:All extensions disabled due to expiration of intermediate... by revvx on Saturday May 04, 2019 @02:39AM (#19824834)
Nope, content marketing company
Re:All extensions disabled due to expiration of intermediate... by brryant on Saturday May 04, 2019 @10:57AM (#19826811)
Not webflow. We auto renew way before LE expires the cert.
Re:All extensions disabled due to expiration of intermediate... by owaislone on Saturday May 04, 2019 @02:00AM (#19824729)
They didn't have renew automatically but they could automate notifications, alerts or even banners in their internal apps when 60-70% of the time was exhausted. If I was given such a restriction, I'd still automate it 100% but require a human to authorize it every time by clicking a magic link in their email, slack or some dashboard, and nag them with notifications until someone authorized it.
Re:All extensions disabled due to expiration of intermediate... by Abishek_Muthian on Friday May 03, 2019 @11:33PM (#19824264)
Some shared hosting like Bluehost now provide LetsEncrypt by default for all their sites with auto-renewal (But I don't recommend Bluehost shared plans for anything even closer to serious hobby due to absurd downtimes like most other shared hosting).

I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.

I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.

Re:All extensions disabled due to expiration of intermediate... by revvx on Friday May 03, 2019 @11:50PM (#19824323)
Another option is using a Web Server/Reverse Proxy that supports Let's Encrypt automatically, like Caddy [1]. I believe Apache HTTPD has partial support [2], too.

[1] https://caddyserver.com

[2] https://httpd.apache.org/docs/2.4/mod/mod_md.html

Re:All extensions disabled due to expiration of intermediate... by glitchcrab on Saturday May 04, 2019 @01:20AM (#19824619)
Traefik is another option here for a reverse proxy with automated renewals; I use it in a ton of places.

https://traefik.io

Re:All extensions disabled due to expiration of intermediate... by Abishek_Muthian on Saturday May 04, 2019 @12:21AM (#19824418)
Apache HTTPD looks interesting, so using which we renew LetsEncrypt cert without using certbot?
Re:All extensions disabled due to expiration of intermediate... by revvx on Saturday May 04, 2019 @12:29AM (#19824442)
It requires some fiddling and it's in experimental state, but yes! Here's the documentation:

https://github.com/icing/mod_md/wiki/Migration

Re:All extensions disabled due to expiration of intermediate... by nsomaru on Saturday May 04, 2019 @12:07AM (#19824384)
Nginx works well and there's a tool that automates most of the extra config stuff for you.
Re:All extensions disabled due to expiration of intermediate... by schwurb on Saturday May 04, 2019 @03:44AM (#19825046)
> had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.

Another use case for the app I am developing! The basic idea: You can enter an item (i.e. "MyOwnShop Cert") into the list. From that time on, it will be tracked how much time passed since the item was entered or renewed (by clicking the renew button). The item with the longest time since entering/renewing is at the top of the list.

Compared to schedules and reminders it has the advantage that the item is not out of our mind once the reminder or schedule pasts. It just sits there dutifully and its timer keeps increasing.

I use it for keeping up with middle-term contacts ("Wow, I have not written Carl for 3 weeks?") and health-related issues. Logging in stuff that easily spoils would be another use case. And, apparently, cert renewals :)

Re:All extensions disabled due to expiration of intermediate... by king_phil on Saturday May 04, 2019 @03:18AM (#19824971)
I own a webhosting provider. We offer Let's Encrypt with automatic issuing and renewal, securing 184,961 hostnames (SANs) at this moment.

We issue certificates automatically if none is existing when connecting to a website and renew the certificates in batches 30 days before they expire. When renewing, we merge certificates/hostnames into bigger certificates with 90 hostnames so we don't have so many moving parts.

If renewal would break, however (as it did once or twice before), nothing bad would happen because on page load there would be a new certificate issued.

Re:All extensions disabled due to expiration of intermediate... by mc32 on Friday May 03, 2019 @10:50PM (#19824056)
So did they conclude it wasn’t a security concern or did they conclude the security risk was worth the uptime?
Re:All extensions disabled due to expiration of intermediate... by revvx on Friday May 03, 2019 @11:32PM (#19824258)
When pressed, they admitted it was just "gut feeling". The team audited a couple ACME clients and couldn't find anything to justify not automating.
Re:All extensions disabled due to expiration of intermediate... by dingaling on Saturday May 04, 2019 @01:12AM (#19824591)
Having a root process with write-privileges to /etc on production machines and also able to communicate over the Internet definitely is a security risk.

To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.

Re:All extensions disabled due to expiration of intermediate... by Whitestrake on Saturday May 04, 2019 @02:09AM (#19824753)
Why would a process need to run as root or have write privileges to /etc in order to automate LetsEncrypt renewals?

I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.

Re:All extensions disabled due to expiration of intermediate... by tedunangst on Saturday May 04, 2019 @02:48AM (#19824866)
Depends on setup, but frequently private keys are inaccessible to the web server worker process. (Which starts as root, loads keys, drops privs, etc.)
Re:All extensions disabled due to expiration of intermediate... by tialaramex on Saturday May 04, 2019 @03:36AM (#19825026)
Most popular ACME (Let's Encrypt) clients allow you to provide a CSR instead of generating the keys themselves. That means a bunch more work for you, but if you're worried about this, that's what you should do. Have your safe (even manual if you insist) process make keys, make CSRs for the keys, and put those somewhere readable. The ACME client will hand them over to the CA saying "I want certs corresponding to these CSRs" without needing access to your TLS private keys at all.
Re:All extensions disabled due to expiration of intermediate... by rocqua on Saturday May 04, 2019 @03:39AM (#19825034)
That does mean you aren't automatically rotating keys anymore.
Re:All extensions disabled due to expiration of intermediate... by revvx on Saturday May 04, 2019 @09:25AM (#19826292)
If you trust your automation, you put private key rotation into it.

If you don't trust it your automation, you rotate the keys manually, as you would normally.

There are no valid reasons to throw the baby away with the bathwater.

Re:All extensions disabled due to expiration of intermediate... by rocqua on Saturday May 04, 2019 @03:38AM (#19825032)
Using http renewal requires listening on port 80 which, by default, requires root.
Re:All extensions disabled due to expiration of intermediate... by Whitestrake on Saturday May 04, 2019 @04:03AM (#19825104)
This is technically true, but contextually lacking.

acme-go/lego doesn't use HTTP validation unless you disable just about every other form of validation first. TLS-ALPN validation is much more likely, so port 443.

That said, it is very easy to allow software to bind to privileged ports without providing it root access; this has been solved for a very, very long time.

Re:All extensions disabled due to expiration of intermediate... by revvx on Saturday May 04, 2019 @08:53AM (#19826112)
You can just use the web server that is already running on the machine.

You (normally) don't want downtime in your website, so you just let your regular webserver serve the acme challenge instead of stopping it.

Re:All extensions disabled due to expiration of intermediate... by mehrdadn on Friday May 03, 2019 @10:58PM (#19824094)
I'm curious as well. My intuition would be that it's not a concern, since servers already keep their private keys stored locally in order to be able to communicate with clients anyway? Being able to update them doesn't really seem to make things any different. But I feel like I could be missing something/not have thought through it properly. (I imagine security implications can get more complicated if a different server decrypts traffic vs. processes it, etc.)
Re:All extensions disabled due to expiration of intermediate... by revvx on Friday May 03, 2019 @11:55PM (#19824339)
The "manual" process used previously by the company already involved some form of automation, so it was more about trusting CertBot not to do anything horrendous.

But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.

Re:All extensions disabled due to expiration of intermediate... by inflatableDodo on Saturday May 04, 2019 @01:49AM (#19824703)
We could attempt a summoning. Quick, make a wildly inaccurate claim about the correct way to implement an encryption library.
Re:All extensions disabled due to expiration of intermediate... by pmontra on Saturday May 04, 2019 @03:04AM (#19824926)
It's automated but things can go wrong even when correctly configured and tested. Real world example: certbot version got old, the renewal server didn't support it anymore, the certificate didn't renew, the web site got the dreaded https warning page.

Of course that is also a kind of misconfiguration. The site has Debian security auto updates on but certbot is not among them. It should be forced to be updated. Furthermore there was no monitoring of errors in its log file.

Still it's not as simple as one believes Letsencrypt to be.

Re:All extensions disabled due to expiration of intermediate... by bluejay2387 on Saturday May 04, 2019 @07:33AM (#19825826)
We update automatically AND manually check periodically to make sure the update took place. That company must be overly fond of drama...
Re:All extensions disabled due to expiration of intermediate... by magicalhippo on Saturday May 04, 2019 @05:40AM (#19825434)
Then the automatic update process stops for some reason and your certificate expires...

At the end of the day, someone needs to verify that new certificates gets acquired and installed before the old ones expire. Automation makes acquiring them less tedious, but not much for making sure someone pays attention.

Re:All extensions disabled due to expiration of intermediate... by djhaskin987 on Saturday May 04, 2019 @09:09AM (#19826191)
Didn't Mozilla invent Let's encrypt? That would make this disaster doubly embarrassing.