I had to apply them per-vhost when using Apache. I switched to caddy recently, which does it automatically for every vhost.

What's caddy? I had the same problem and just ended up separating all my SSL vhosts into their own files.

You're using caddy in production? Any comments on that?

It's the fastest and most stable web server I've ever used. It does crash whenever something is misconfigured, but that is usually my fault. It works wonders and never crashes unless I cause it to.

I was thinking of adding it to my home box, but I'm afraid of any vulnerabilities, since it's not very mature yet.

I've read somewhere that it won't work if there are multiple VirtualHost blocks in the same file. Allegedly if you separate each vhost config into it's own file, it will start working. YMMV.

I just tried it, seems to work fine.