News for hackers. Stuff that matters.
Re:Let's Encrypt is down by discreditable on Friday May 19, 2017 @09:04AM (#14375175)
Wow @ that close comment:

> So, this is not a bug and all is working as intended.

Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!

Re:Let's Encrypt is down by tyingq on Friday May 19, 2017 @09:10AM (#14375228)
That's going to be a limiter for adoption. Hopefully @mholt reconsiders.

Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...

Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @09:24AM (#14375351)
Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx.

Having the server be unable to start through circumstances outside of the system's control is just such a huge no.

Re:Let's Encrypt is down by danielbln on Friday May 19, 2017 @09:33AM (#14375434)
Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far.

edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.

Re:Let's Encrypt is down by discreditable on Friday May 19, 2017 @11:46AM (#14376791)
Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
Re:Let's Encrypt is down by enimodas on Friday May 19, 2017 @09:51AM (#14375594)
Why did you switch away from nginx?
Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @10:12AM (#14375800)
Not having to deal with certificate renewal is a big deal.
Re:Let's Encrypt is down by majewsky on Friday May 19, 2017 @10:28AM (#14375951)
Is it? I spent one day getting certbot up and running, and ever since then it's been pretty much a done deal.
Re:Let's Encrypt is down by manigandham on Friday May 19, 2017 @05:28PM (#14379487)
That's 1 day that I didn't have to spend, even better when dealing with many instances.
Re:Let's Encrypt is down by majewsky on Saturday May 20, 2017 @08:01AM (#14382011)
Okay, I agree, with two caveats:

1. The cost does not scale with the amount of instances since it is the one-time cost to create the configuration package.

2. If you decide to go for Caddy instead, you'll have to spend the same time, if not more, learning Caddy.

Re:Let's Encrypt is down by fb03 on Friday May 19, 2017 @10:26AM (#14375927)
I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy.

Am I missing something?

Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @10:32AM (#14375996)
And by running Caddy instead, I have one less piece to monitor and worry about.

The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

Re:Let's Encrypt is down by stephenr on Friday May 19, 2017 @10:48AM (#14376153)
> by running Caddy instead, I have one less piece to monitor and worry about

No, with Caddy you have several vaguely related pieces glued together with superglue.

> The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

I think this thread is pretty much proof that that approach will bite you in the ass.

Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @11:01AM (#14376318)
> I think this thread is pretty much proof that that approach will bite you in the ass.

What bit me here is the fact that I'm running alpha software instead of a battle-tested web server; I'm doing so willingly, with full awareness of the risks that that entails.

Drawing the conclusion you did from the variables at play is shortsighted. If anything bites people in the ass, it's prejudice and shortsightedness. I wouldn't want you handling my ops/infrastructure.

Re:Let's Encrypt is down by stephenr on Friday May 19, 2017 @11:22AM (#14376543)
> What bit me here is the fact that I'm running alpha software

This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were <= 30 days away from expiring and the CA can't be contacted.

> Drawing the conclusion you did from the variables at play is shortsighted

Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't working the way you want, you have to either a) replace it completely or b) work out how to disable the bit that's not working how you want, and replace that part of it.

> Drawing the conclusion you did from the variables at play is shortsighted

- People use a piece of software that serves as both ACME TLS certificate client and web server

- Said software by design won't start if the CA can't be contacted 30-days out from expiry

The conclusion I drew is that such integration leave the operator with less control than if they followed a separation-of-concerns approach, and left web serving to a web server, and TLS certificate renewal to an ACME client. The former doesn't need to care about how old the certificates are, just use what it's given.

Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @11:30AM (#14376619)
You're drawing conclusions from unintended behaviour, which has now changed (and a release has been issued).
Re:Let's Encrypt is down by gommm on Friday May 19, 2017 @01:48PM (#14377899)
I still think that refusing to start if the cert expires in 7 days or less is still an issue if Let's Encrypt is down.

There should be at most a warning but it should start. Otherwise you end up with an external dependency that can cause your web server to not start through no fault of your own.

Re:Let's Encrypt is down by stephenr on Friday May 19, 2017 @02:01PM (#14378000)
> unintended behaviour

Ahem. https://github.com/mholt/caddy/issues/1680#issuecomment-3026...

Emphasis mine:

> So, this is not a bug and all is working as intended.

Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @08:46PM (#14380386)
Have you noticed that the bug has been fixed?
Re:Let's Encrypt is down by stephenr on Friday May 19, 2017 @11:02PM (#14380835)
why do you keep calling it a bug?
Re:Let's Encrypt is down by JshWright on Friday May 19, 2017 @10:38AM (#14376042)
> The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

That's an odd thing to say in this particular conversation thread...

Why would you want to tightly couple your webserver to the availability of another service provider?

Re:Let's Encrypt is down by scrollaway on Friday May 19, 2017 @10:47AM (#14376130)
It's not tightly coupled any more than it would be with certbot. The issue I filed is an issue because it's unexpected behaviour (despite early closure, it has actually now been fixed).
Re:Let's Encrypt is down by tyingq on Friday May 19, 2017 @10:42AM (#14376075)
The simple config syntax and sane defaults is nice in Caddy. For example, 3 lines of config nets a https server with http/2 support and an A rating with Qualsys for ssl setup.
Re:Let's Encrypt is down by stephenr on Friday May 19, 2017 @02:02PM (#14378012)
> sane defaults

Failure to start if a CA is down is a "sane default" ?

Re:Let's Encrypt is down by tyingq on Friday May 19, 2017 @02:13PM (#14378087)
Yes, that's exactly what I meant. You can tell because I didn't present any examples. And because one issue invalidates everything else. And I didn't express my opinion on this issue up thread.
Re:Let's Encrypt is down by eridius on Friday May 19, 2017 @04:53PM (#14379274)
I switched from nginx to caddy for my (extremely low-traffic) server because I was tired of having to copy & paste a bunch of SSL setup any time I was configuring a new domain.
Re:Let's Encrypt is down by GiorgioG on Friday May 19, 2017 @10:00AM (#14375679)
I'm done with Caddy - this is security theater at its finest. Back to nginx we go. Why did I pick Caddy, because it was simple/easy/fast-to-setup.
Re:Let's Encrypt is down by e_d_e_v on Friday May 19, 2017 @10:40AM (#14376066)
I moved from Caddy to Traefik (https://traefik.io/) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.
Re:Let's Encrypt is down by mholt on Friday May 19, 2017 @11:38AM (#14376711)
Re:Let's Encrypt is down by e_d_e_v on Friday May 19, 2017 @02:31PM (#14378233)
Nope.
Re:Let's Encrypt is down by merkaloid on Friday May 19, 2017 @10:27AM (#14375939)
Or someone forks it
Re:Let's Encrypt is down by kuschku on Friday May 19, 2017 @11:41AM (#14376743)
Then wait until you see how caddy handles fully qualified domain names!

Caddy will just refuse to even handle them.

Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended.

Try out: https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ redirects to the relative domain

and then https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ )

Re:Let's Encrypt is down by yjftsjthsd-h on Friday May 19, 2017 @12:23PM (#14377147)
Is there a bug for this? That's out of spec
Re:Let's Encrypt is down by kuschku on Friday May 19, 2017 @01:00PM (#14377493)
Yes there is, @mholt said it’s a WONTFIX. You're supposed to script it yourself.

https://github.com/mholt/caddy/issues/1632

Re:Let's Encrypt is down by Moter8 on Friday May 19, 2017 @03:24PM (#14378670)
I submitted the issue for it (https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.
Re:Let's Encrypt is down by Vendan on Friday May 19, 2017 @09:05AM (#14375181)
I clicked the link, saw it was closed, and thought, "Wow, these guys are fast". Then I read it....
Re:Let's Encrypt is down by mholt on Friday May 19, 2017 @09:57AM (#14375655)
Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1
Re:Let's Encrypt is down by atonse on Friday May 19, 2017 @10:26AM (#14375923)
This misses the overall point.

If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes.

This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?

Re:Let's Encrypt is down by majewsky on Friday May 19, 2017 @10:29AM (#14375962)
> If I am hosting 5 sites on caddy, and add a 6th one, I restart the server.

No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)

Re:Let's Encrypt is down by sleepychu on Friday May 19, 2017 @10:42AM (#14376078)
OK, there's a 0-day patch for OpenSSL, without it my users are vulnerable to RCE. Why can't I restart if LE's ACME is down?
Re:Let's Encrypt is down by e_d_e_v on Friday May 19, 2017 @10:38AM (#14376043)
This goes wholly against most cattle-not-pets devops philosophy though, right?
Re:Let's Encrypt is down by eropple on Friday May 19, 2017 @10:48AM (#14376144)
How so? In Chef:

   notifies :reload, "service[caddy]", :delayed
Re:Let's Encrypt is down by e_d_e_v on Friday May 19, 2017 @11:14AM (#14376461)
The idea here is that the server can't be scaled up or down. I suggest googling "cattle vs pets". If you know how to make a single process scale horizontally across additional hosts or scale up in alternative datacenters with a chef service notification, I'd pay you money to tell me how.
Re:Let's Encrypt is down by eropple on Friday May 19, 2017 @01:39PM (#14377835)
So, um: devops and cloud architecture is my job. I think I know what "cattle vs pets" is referring to. Nothing in this thread has anything to do with one versus many processes or one versus many nodes, nor does it have anything to do with manual configuration of anything. Rewrite the configuration based in orchestration data or, in extremity, upload a new version of a cookbook that handles that new site; the hypothetical sixth site is added and the service kicked over without human interaction. Scaling is an orthogonal concern.
Re:Let's Encrypt is down by ptman on Friday May 19, 2017 @10:19AM (#14375867)
What if there's a reason to restart the server or spawn a new one?
Re:Let's Encrypt is down by imhoguy on Friday May 19, 2017 @12:41PM (#14377309)
Scaling, upgrading, disk issues, just migrating to new place... and foremost https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_com... Also nowadays with Docker around I rarely see server/app bundles supporting graceful reload.