Automation failing once shouldn't be a catastrophe; you could renew after 30 days and issue a pager alert if it fails. Then you'd have 60 days to figure out what went wrong, manually update it, and fix the issue with the automation.

At fully loaded engineer costs, if someone takes more than an hour or two to investigate, its already more expensive than a wildcard cert.

The few times I've had Let's Encrypt's automation fail it's taken a few minutes, not hours, to investigate and fix. Anecdotal, sure, but I don't see it taking hours to fix

Anecdotal, but managing thousands of VMs, hundreds of ELBs, service discovery, several CI pipelines, and so on, its never a few minutes unless its a blatant misconfigured flag.

It sounds as if your organization would be a customer for a different service than the one that LE provides.

If you're issuing lots of internal certificates, you may be interested in using Anchor [1] instead. It's designed for that use case and with own service you don't need to worry about rate limits, name limits of similar issues.

While it's designed to cycle all your certificates every day, you can still configure it for weeks/months. It's also based on credentials rather than domain verification (shameless plug, I'm Anchor developer)

[1] https://github.com/openstack/anchor

Apples/oranges comparison.

You're comparing engineer time to the wildcard cert transaction price. You're neglecting to include the time for dealing with the wildcard cert (including manually remembering to renew that).

With process automation, you (or better: a collaboratively supported tool) gets progressively better at nailing renewals.

JoshTriplett's "renew every 30 days, 60 days to fix the problem" approach strikes me as inspired.

How much does it cost to manually renew your certificate every year? How much does it cost if you forget to manually renew it, or make a mistake with one of the manual steps?

This is only true if all of your engineers are operating at 100% capacity all the time. In my experience that's generally not true (and if it was they'd probably burn out pretty fast). Unless you're staffed entirely by contractors and pay by the hour?

If all you care about is money, sure.

> The chances of automation failing is higher than the chance our SSL certs will need revocation.

Given a 90-day duration, set up automated renewal for every 30 days, and if that renewal fails, you'll have 60 days to deal with it.

Or just do what Let's Encrypt suggests and have a cron job that runs certbot-auto's renewal command twice a day, every day? It doesn't actually renew until nearing the end of the 3 month cycle, but by having two attempts per day it will make it far more likely to go through if the network is down or something otherwise prevents it from renewing one of the times. They, of course, also will email you repeatedly as you near the end if you haven't yet renewed.

Sounds like a great strategy!

If critical automation failure is treated as a serious bug in LE, then it should really just be a matter of time before 99.9% of them are fixed and LE can enter a state of maintenance rather than development. LE (the client) don't need to expand until it can read mail.