If you want to be super minimal, I prefer acme.sh[1] instead. It even comes preconfigured for various DNS providers[2], and you can even create your own hook if there isn't already one[3].

[1] https://github.com/acmesh-official/acme.sh

[2] https://github.com/acmesh-official/acme.sh/wiki/dnsapi

[3] https://github.com/acmesh-official/acme.sh/wiki/DNS-API-Dev-...

If you have over a thousand lines of bash (or any other kind of shell code really), that is a pretty big red flag that you probably shouldn't be using bash IMO.

If you want to issue a certificate in an environment where you only have a shell, it's the only language you can use.

Sometimes you write in shell because it's the lowest common denominator.

Well, if you have to use shell that's one thing, but I would be hesitant to use such a large shell script for something as important as certificate issuance in an environment where I didn't have to.

Just as a counter point, I've been using acme.sh for ~3 years now and it's been rock solid.

I get your point, and was pretty shocked to find acme.sh, but after the certbot PPA made a giant mess of my system I gave it a try. On the other hand, why should I balk at running thousands of lines of bash, but be fine with thousands (or many more) lines of C, Python, PERL...? You can write crappy or beautiful code in any language...

> If you only have shell on your servers then it is time to start looking for a new job.

Perhaps you wish to have "real" certs on appliances like F5s and Isilons (FreeBSD-based) where you can't install extra stuff, but where curl and openssl (and bash/zsh) are present.

Or perhaps you want to run simple software that you can actually audit. While "over a thousand lines of bash" may take a little while to examine, good luck auditing Zope, which is what certbot pulls in as a dependency:

* https://packages.debian.org/bullseye/python3-certbot

* https://packages.ubuntu.com/jammy/python3-certbot

Here we are talking about a lightweight C executable though that doesn't have those dependencies. You are also not limited to provisioning certificates on appliances as well, and in those cases I don't think have a thousand line bash script offers anymore security (probably less) than a full-featured C program.

Except you have to be able to compile the C code into an executable, which may not be possible on an appliance.

appliance as in a router or a washing machine? who, ever, would not pre-compile and distribute binaries in a situation like that?

At my last job I ran an Isilon: I could upload a cert for the HTTP server via the web UI, but there was no ACME client. I could SSH in, drop dehydrated and have it work because all I needed was a shell, curl, and openssl.

Similarly with F5: there is (was?) no native ACME client (at least a few years ago when I first looked at it). So I download dehydrated and used various CLI interfaces to schedule automated runs and importation of the certificates.

There was no pre-compiled binary, and no compilers, on either system, and so talking about a "lightweight C executable" is non-sensical. Further, even if we (managed to) compiled things off-host, when we did an OS upgrade on either system, a whole bunch of libraries would change and we'd have to (remember to) re-compile. There is no such worry with a shell script.

If you want to have ACME-fetched certs on a general computer system, then compiling a C program (large or small) is an option. But there are scenarios where compiled/compiling C programs is not an option, and you telling me otherwise when I have personal experience of these situations takes some chutzpah.

I wouldn't be offended. Many people including me have personal and work experience in this area as well. No one is saying you're wrong, but even you acknowledge there are other ways to upload certificates.. usually involving an API as well. If you want to run unchecked third-party 1000+ line bash scripts on production appliances, by all means go right ahead.

> If you want to run unchecked third-party 1000+ line bash scripts on production appliances, by all means go right ahead.

Again, I have a better chance at reading all the code of dehydrated (which I have, in fact, done), then reading all of the Python code that certbot pulls in via dependencies on Ubuntu/Debian.

If you’re provisioning an immutable VM or a container you don’t want to add unnecessary cruft. The official LetsEncrypt client and its > 100 dependencies is a non-starter.

Not everything I do is a job.

Stop bashing on bash.

It has limitations, and quoting takes a hot minute to grok, but those don't come into play for a surprising amount of medium-large projects when used properly.

I use POSIX sh only and get by with maintainability and handling failure modes just fine.

Bash is great for cases where you are gluing together a bunch of other commands. But it also has a lot of pitfalls, that something as large and complex as this will absolutely run into (to be fair, so does c).

As a specific example, the ACME protocol requires working with json. Doing this in bash is very difficult and error prone, especially if you want to avoid a dependency on something like jq, as this does.

ShellCheck and unofficial strict mode make bash a pretty nice language.

On the otherhand, if you're a user and not a developer, you know something written in bash will be written to run on any machine out there. Doesn't matter if it's old or new. Whereas if it's written in C++xx or Rust or the like it'll only compile/run on rolling release distros (or for the 3 months after a normal distro is released that it's up to date).

It might only compile on a machine with a recent compiler if it’s aggressive with using new language features, but why would need to compile a Rust/C++ version from source? A compiler binary will run jut fine on old distro versions.

On old distros you need to install the latest libc and libstdc++/libc++ to run them. I’ve tried many times in the past and never succeeded.

Typically projects will compile the binaries on systems with a much older version of libc (but the latest compiler) specifically to avoid this problem.

If they’re not doing this then I believe they won’t work on older systems even if compiled with old compiler versions.

Bash isn’t on non-UNIX machines by default. I could run a C or Rust produced binary just fine, but I couldn’t run that shell script.

Minimalism is much different than portability. I don’t consider heavy bash scripts to be minimal.

I use this and it's pretty great.

Also it can be highly locked down - run as it's own unprivileged user, with access only to directories served by another webserver for the ACME handshake, storing certs, and a tightly restricted sudoer to restart the webserver on cert cycle.

> It even comes preconfigured for various DNS providers[2]

Also, CLI utility that supports a bunch of APIs:

* https://github.com/AnalogJ/lexicon

I found this project while looking for a way to renew my SSL certificate without having to use certbot which has a lot of dependencies including python. This program is really small and simple and does exactly what I need. It's perfect.

If you like minimal dependencies another one to take a peek at may be acme.sh [1]. It depends on bash, openssl and curl. It seems to work fine in ash as well. It has code to handle most API's and most importantly to me is the great documentation.

In the same spirit of minimal and light weight there is also testssh.sh [2] for testing TLS on HTTPS/SMTPS servers that also depends on bash and openssl.

[1] - https://github.com/acmesh-official/acme.sh

[2] - https://github.com/drwetter/testssl.sh

I'd prefer to use a C, Go, or Rust app at this point. I love shell scripts because it was one of the first scripting languages I learned, but I'd trust a developer capable of writing C, Go or Rust to do a better job and make something more optimized than what is within the scope of Posix shell scripting.

There's step-cli that's just a single go executable: https://github.com/smallstep/cli

They also provide a CA to go with it, if you need internal certs.

Thanks. I actually have this installed. It does so many things and is pretty incredible!

To me it is a utility stuff. As long as it does the job (and acme.sh does it just fine) and does not require pulling down half of the Internet for dependencies I would not give rat's ass about what language has been used to write it.

I don't think optimization really matters to most people for a tool that does automated cert requests.

It is to an extent. I'm not saying acme.sh is bad just that if there is a tool in Go, Rust or C that does the same thing and is more efficient then I'm picking something that isn't wrapped in a bunch of shell code. Same with tiny webservers.

I use https://github.com/go-acme/lego, single Go binary.

Personally, I'm a fan of https://github.com/diafygi/acme-tiny. 200 lines of python without any additional python requirements and only the openssl binary as external dependency.

If we're plugging implementations, I tend to use the single-file implementation that ships with 9front. I wrote the first cut, but it's been improved heavily by others:

http://man.9front.org/8/acmed

http://git.9front.org/plan9front/plan9front/HEAD/sys/src/cmd...

It works quite well, and if you mount your unix machines via sshfs, it's pretty easy to dump the certs into the right place with them.

And unlike most ACME clients, it works seamlessly with DNS authentication, which allows you to easily grab wildcard certs.

This! Thanks for also mentioning it. So plain easy and just does what I need! Thanks to the author for publishing it.

I maintain my own patch, so tiny-acme supports an '--outfile' option (it originally only writes to stdout). This comes in handy when it is run by systemd service/timer.

The pull request is on hold, because the code then exceeds then 200 lines threshold :shrug:

You can directly redirect the service's output to a file: https://www.freedesktop.org/software/systemd/man/systemd.exe...

Thanks! That's tempting to drop the patch and just stay with the upstream.

I will plug my own Python program https://github.com/frutiger/concorde, which does depend on `requests` and `cryptography`.

You can use Caddy to automatically fetch certs and keep them renewed without much hassle[0].

Disclaimer: I'm affiliated with Caddy

[0]: https://caddy.community/t/using-caddy-to-keep-certificates-r...

Yeah IMHO this is the way to go. Individual web apps managing their own SSL certs is a longterm mess. Only your proxy or HTTP gateway/router should ever touch or know about SSL certs.

Caddy is great for web servers, but it's still not possible to have it run commands post certificate provisioning. So it's kind of a non-starter for anything but web-servers as there is no way to tell a different system to reload certs.

We're working on this! Hoping to have our new event dispatching system ready in the next few months. This'll let you hook into the post-issuance event and do whatever you want afterwards.

Yeah! I saw the PR. Looking forward!

Not affiliated with caddy, but caddy is awesome! I use it all the time.

Apache also provides a 1-at party ACME client in the form of a module, mod_md.

I'd rater use Nginx. At least it's not a statically linked blob.

Static linking is a huge advantage. I don't understand the point you're trying to make.

Static linking is a huge disadvantage when it comes do security updates.

C is a huge disadvantage when it comes do security.

No, that depends entirely on how it's written.

I'm using lego - https://github.com/go-acme/lego for this, single binary works great.

This is the joy of static linking!

In the end does anyone really care if it’s a 10MB tarball versus a 330kb tarball?

Yes, many of us care.

Yes, very much so. Maintaining the security of OSes where many applications statically link thousands of dependencies is impossible.

You care when you need to deploy stuff to remote servers the time difference between 10MB and 330kb. There’s also data transfer costs to consider.

I never had a problem with how long it took to install dependencies until I had to do it multiple times over in quick succession.

Why wouldn't you upload the file to all the servers, and then deploy them all in succession, rather than upload and deploy in sequence?

Dehydrated is good for this, too https://dehydrated.io/

I used dehydrated pretty effectively (along with openssl) to renew and sync certs between several layers of proxies/loadbalancers. I ended up creating a nice k8s deployment with CronJob to implement this with ingress-nginx.

https://man.openbsd.org/acme-client.1 is similar. I'm not sure if it was ported to Linux.

It was, I maintain a port here [0]. Release tarballs can be downloaded here [1], I plan to add them to the git tags but did not get to it yet. For example alpine linux ships it in the repositories [2].

0: https://git.sr.ht/~graywolf/acme-client-portable

1: https://data.wolfsden.cz/sources/

2: https://pkgs.alpinelinux.org/packages?name=acme-client&branc...

I've recently spent some time sandboxing nginx on my web server by running the master process as the user www-data instead of root, adding some seccomp filters through systemd and writing a custom AppArmor profile[1]. And the hardest part was dealing with certbot actually. Not least because it's written in Python, and so adding capabilities to the script itself didn't do a thing. They needed to be added to the Python interpreter executable[2]. The whole thing seemed more complicated than its worth. I've looked briefly at what supposedly configuring an OpenBSD server with OpenBSD httpd + OpenBSD's acme-client would look like, and it seems so much easier and the secure configuration is the official one, so I'd need to worry much less about updates breaking my stuff.

So I'm considering moving to OpenBSD for my web server. The only thing giving me pause currently is that I saw no mention of HTTP/2 support, so I suspect there is none. Although maybe I don't need it. We'll see.

[1]: That last apart I actually already had, but it needed some tweaking this time, because I moved nginx's PID file to its own runtime directory created by systemd, instead of allowing it to write to /run as root.

[2]: Of course only for testing. I revoked them after verifying that everything works. And now they're only assigned dynamically by systemd when the service is run.

Quite reliable as well. I haven't touched my acme-client.conf since 2019.

Admittedly, I haven't updated my blog since then, but goddammit, my SSL certs are still valid! Cough Manjaro Cough

One of the things I don't like about Github is that, while they do list languages used, they don't have a section for listing dependencies. Some (Ok, many) projects don't list dependencies and assume we're all OK with so many that we don't deserve to know and are expected to just pipe a URL to a shell to install them all.

We need more projects like this, because many of us would like clean, reproducible environments that won't be in dependency hell every few years when an update to one dependency isn't compatible with updates to others.

Certificates seem to be lots of work and complexity, and lots of do-it-yourself. Couldn't that functionality come as part of the OS?

It is fundamental that computers should be able to know whom to trust and whom not.

It is in windows, the cert store comes with windows and you use for many applications from code signing, email signing, server cert signing to authenticating users and the computer to the network/AD.

PKI is the way windows and the corporate world went. On nix you have PGP, with GPG it is sort of baked in I guess but it is used only in package signing and signing of text content in various contexts.

I believe the "web of trust" is preferred over PKI but nix folks because they don't want to depend on certificate authorities mediating trust, instead you have key servers that only help in distributing keys. For *nix, the burden of forming trust is on the user. You have to either go to key signing parties (very scalable lol) or I guess google around and ask in chatrooms and look up trustworthy sites until you are confident a key us trustworthy? I mean you can cross-sign and all that but in reality, you have to investigate each key one by one and hope you don't screw up. Or what happens in most cases is people trusr https/PKI so key IDs on people's websites or a command asking you to import a key from their site over https is trusted... because PKI/TLS is much more practical. Get the GPG fans to like PKI and maybe Linux will have PKI cert management out of the box. The Linux kernel does support X509 and pki certs afaik, I don't know if you can make it store root certs and have it verify cert chains for you from userspace. Very possible but only applications using kTLS might want that. Perhaps a feature request with your favorite browsers and package managers to use kTLS instead of openssl/gnutls is a start?

I suspect there isn't a Windows Enterprise PKI team any more as such, development seems to have stalled around 2016.

There is no ACME support built-in either as a client or server, which is sad considering Windows already has dynamic DNS capabilities.

Even Azure Key Vault won't integrate with Microsoft Enterprise PKI, which should tell you everything you need to know...

On prem AD itself is going away, no idea if AAD has or will have a PKI component.

Cool! I hadn't seen tls-alpn-01 for authentication before.

Instead of using the ualpn daemon to respond to the challenges and proxying all other connections through to nginx, would it be possible to do it solely in nginx?

I really appreciate the limited dependencies. The readme says it needs only "libcurl and one of GnuTLS, OpenSSL or mbedTLS"

Been ussing this script https://github.com/srvrco/getssl

Someone needs to tell the Manjaro folks about this.

Security software written in C, with no unit tests. You cannot run away from this software fast enough. I cannot think of any worse idea than "I wrote my own base64 codec in bare C without tests and without code review". "Minimal dependencies" does not even begin to make up for how bad this idea is. It would be strongly preferable to depend on third-party code that has been reviewed, tested, and implemented in a reasonable language.

You use C and C++ apps daily— many of which probably have poorer automated testing than you’d think. It’s not great, but it isn’t the world-ending catastrophe that you seem to think it is.

he is spot on, you know - this software is a contender for foundational infrastructure. The best foundational infrastructure software is like SQLITE - more tests than code, tiny and runs everywhere

And written in C. So, he is at best half spot on. I agree that I’d want to see better testing. I would disagree that C rules it out.

No, I want my Debian machine polluted with Snap garbage to run Certbot. Find someone sane to promote this to. :)

Right? It’s so ridiculous how you’re supposed to use Snap to install certbot. The (well, one of..) GitHub discussion is just beyond the pale:

https://github.com/certbot/certbot/issues/8345#issuecomment-...

I really hate how all that junk urinates into /etc, which should be treated as immutable by applications. Changing state should go into /var/run.

/var/run isn't really suitable for certificates, as it's allowed to get cleared on every boot. /var/lib would be the better alternative.

Just install docker or k8s and use certbot image.

I know you jest, but if you're in k8s, I'd check out cert-manager (https://cert-manager.io/). It works quite well, and as it integrates w/ k8s, it stores the cert in a Secret where an Ingress picks it up automatically, and it solves the whole multiple-replicas-all-need-the-cert problem that I have w/ certbot.

(I agree, while certbot works, it's a bit of a usability nightmare.)

Wish it had a way to kick off a pod after a certificate is issued, that way I could automate applying the issued certificates to the devices where they'll be used.

(Yeah I currently do this with a cron job so it's not _so_ bad)...

Yes. I would appreciate such a feature, too. For Ingresses, it doesn't matter (as the controller is smart & picks it up automatically) but we have a database whose cert requires a rolling restart, and yeah, it'd be good to have a way to trigger that.

(We get around this, normally, by having regular reboots of the nodes for system updates. So no pod lives long enough for a replacement cert to not get picked up in time, usually. In the off case reboots don't happen, we have alerting.)

And have docker insert firewall rules in ways you didn't expect on a production server? Sounds fun.

The solution is frequently less software, not more.

Well, podman then? I think it's pretty much user-space and does not tinker with system settings.

The thread above has mentioned a very real problem with snap running on OpenVZ. It requires either squashfs as a kernel module or FUSE+squashfuse. Many OpenVZ hosts will not provide either and docker/k8s is obviously out if you are already operating inside a container...

Like, squashfs is great and all and packaged app images are an acceptable way of delivering desktop apps... but there's so much lossage here when certbot indirectly requires both just for an install.

(I ended up going with acme.sh instead.)

I think vbezhenar was just joking.

Frankly at this point I'd happily accept it if certbot required docker or k8s over snap.

At least then I know I'll have to run it in a container (and give up then) instead of waste hours figuring out why I get a snap error message when I try to run/install certbot before finally coming to the conclusion it is impossible.

Poe's law detected.