Letsencrypt is painful on docker. I dont mind paying 40$ per year for a wildcard ssl certificate.
The only hard part is you'll have to find a way to prove ownership other than email ownership. But if these are public-facing web servers, implementing a response for the HTTP challenge shouldn't be hard, and as the website points out, you can configure all your servers to send an HTTP redirect for the challenge to some other single URL, which you can configure manually. https://letsencrypt.org/docs/integration-guide/#picking-a-ch...
This process is significantly less painful, especially on renewals, than the traditional certificate renewal process. If you're already planning on spending an hour of someone's time a year to request a renewal, pay for it, click a link in an email, etc., plan on spending 10 minute of someone's time every two months, instead. (And an hour is optimistic based on my experience.)
Your public and private keys don't change, since these are renewals. You're just updating the certificates themselves. And certificates are public data (they're sent in their entirety when you make an HTTPS connection), so you can just put them in a git repo or an S3 bucket or whatever else is convenient for your deploy process.
We buy 3 year wildcard certs - Rapidssl was about 80$. From about a month before renewal, they start sending you emails and make sure you don't forget.
Letsencrypt is a good idea for larger setups with dedicated devops. For a start-up, it's the exact same argument as Gitlab vs Github.
There is no reason for these apps to have a public cert. An app shouldn't be publicly exposed.
Why does that matter? Pretty sure you don't have to change your public key to get or renew a Let's Encrypt cert.