News for hackers. Stuff that matters.
Re:Nginx Modern Reference Architectures by nikivi on Thursday May 19, 2022 @12:55PM (#31437096)
I am curious why I should consider using nginx when Caddy (https://caddyserver.com) exists. It seems to have more activity behind it.
Re:Nginx Modern Reference Architectures by corrral on Thursday May 19, 2022 @01:23PM (#31437513)
It's much more widely used and has been around a whole lot longer, both of which confer many benefits.

Personally, if nginx made letsencrypt as easy as caddy does, I'd never even think about caddy. That's its One Thing that kinda tempts me.

Re:Nginx Modern Reference Architectures by ancientsofmumu on Thursday May 19, 2022 @04:41PM (#31439895)
> if nginx made letsencrypt as easy as caddy does

The EFF cerbot plugin has it covered; assuming your generic Debian server with nginx already configured to host www.domain.com (trivial hello, world setup - nothing fancy) then it's:

    apt-get install cerbot python3-certbot-nginx
    
    certbot --nginx -d www.domain.com -d domain.com \
      --agree-tos -m "[email protected]" --no-eff-email \
      --deploy-hook "systemctl reload nginx"
    
    systemctl restart nginx
All done. Certbot is already running as a systemd service to handle ongoing renewals and it'll now restart nginx if your cert is updated. This example uses the trivial http-01 ACME method, if you need the more complex DNS based setup for wildcards that'll take a bit more elbow grease.
Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @04:49PM (#31439976)
FWIW, having the ACME client separate from the server has a bunch of downsides. It's less robust, can't provide OCSP stapling and automatic renewal on revocation, doesn't have issuer fallback, can't offer you On-Demand TLS, etc.
Re:Nginx Modern Reference Architectures by pilif on Thursday May 19, 2022 @05:10PM (#31440209)
I agree with most points, but OCSP stapling is independent of ACME and thus is perfectly doable with nginx and an externally obtained let’s encrypt certificate.

That aside, for me the trade-off was different and I was willing to give up the benefits of included acme support for the benefits of running a very well-supported and well-known web server that at this point hosts most of the internet and which can run on port 80/443 without iptables hacks (not sure whether this still applies to caddy)

Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @05:23PM (#31440339)
What I meant was using OCSP status (from stapling) to trigger reissuance on revocation. I don't think this can be done with nginx and certbot unless nginx makes its OCSP status available for the certbot client to read from, or having an event trigger in nginx somehow to get certbot to run. Either way, it's extra faff that you don't need to worry about with Caddy.

> which can run on port 80/443 without iptables hacks

Not sure what you mean. Do you mean that you need root to bind to those ports? In which case, you can give the process CAP_NET_BIND_SERVICE which lets it. Caddy's systemd service does this, and runs as a non-root user: https://github.com/caddyserver/dist/blob/2ceb535e076ed9b3083...

Re:Nginx Modern Reference Architectures by mholt on Friday May 20, 2022 @12:02PM (#31449265)
Let's Encrypt themselves has called Caddy the "gold standard" of certificate automation. (Not certbot.)
Re:Nginx Modern Reference Architectures by hn_version_0023 on Thursday May 19, 2022 @02:27PM (#31438295)
/me mumbles incoherently about Apache httpd
Re:Nginx Modern Reference Architectures by corrral on Thursday May 19, 2022 @02:30PM (#31438322)
Hahaha, I'm a dinosaur, too. I only started considering nginx something other than "that new-fangled crap" a few years ago. Caddy may get there in about a decade.
Re:Nginx Modern Reference Architectures by aaaaaaaaata on Thursday May 19, 2022 @02:41PM (#31438450)
Just when I think "It's gotta be secure by now, they must have figured it out" there's a new and disturbing Apache CVE — change my view.
Re:Nginx Modern Reference Architectures by Spivak on Thursday May 19, 2022 @04:37PM (#31439850)
httpd has the advantage that it has literally every feature you could imagine and a million more you couldn't but it's a slog to actually use.
Re:Nginx Modern Reference Architectures by anotherevan on Thursday May 19, 2022 @08:25PM (#31441959)
> it's a slog to actually use.

I don't know. I haven't used Apache HTTP in a while, but Nginx seems to have a lot more foot-guns. I'm planning on giving Caddy a go on my next server upgrade.

Re:Nginx Modern Reference Architectures by attentive on Thursday May 19, 2022 @02:37PM (#31438393)
funny enough, apache does have mod_md that will deal with LetsEncrypt
Re:Nginx Modern Reference Architectures by boardwaalk on Thursday May 19, 2022 @01:44PM (#31437798)
For concrete comparison, nginx backs something like 40% of the top 10k websites. Caddy is a fraction of a percent.
Re:Nginx Modern Reference Architectures by mooreds on Thursday May 19, 2022 @02:57PM (#31438620)
Here's the numbers I commonly cite: https://news.netcraft.com/archives/2022/04/27/april-2022-web...

31% nginx 23% apache 8% Openresty (which I've never heard of, looks like it is an augmented nginx) 5% cloudflare

Re:Nginx Modern Reference Architectures by favorited on Thursday May 19, 2022 @04:48PM (#31439964)
Openresty is basically Nginx + Lua. I did some POCs on it back in the day, honestly surprised it's still around. It wasn't bad, just seemed like a product with a very small niche.
Re:Nginx Modern Reference Architectures by mwcampbell on Thursday May 19, 2022 @05:00PM (#31440103)
If I'm not mistaken, the biggest user of openresty is the Kubernetes community-maintained nginx ingress controller [1].

[1]: https://github.com/kubernetes/ingress-nginx I took pains to say "community-maintained" because there's also an official nginx ingress controller from F5 (current corporate owner of nginx).

Re:Nginx Modern Reference Architectures by phillipseamore on Thursday May 19, 2022 @10:15PM (#31442604)
openresty and cloudflare are both nginx forks
Re:Nginx Modern Reference Architectures by the_third_wave on Thursday May 19, 2022 @05:02PM (#31440127)
I never experienced nginx' way of handling certificates as difficult and I prefer to keep my web server independent from the certificate management system since I use those certificates for more purposes than just web servers so I have not looked into the supposedly superior way caddy handles this workflow. I handle certificate requests and updates on a separate system which then distributes certificates and handles daemon restarts to where and when these are needed by using certbot hooks. All it takes to request and distribute a new certificate is a single command on that certificate management system, the rest follows automagically. Am I missing something by doing things this way?
Re:Nginx Modern Reference Architectures by dekobon on Thursday May 19, 2022 @06:36PM (#31441061)
Disclaimer: I am one of the authors of the project.

I do wish that NGINX made LetsEncrypt as easy as to use as Caddy does. We are all big fans of LetsEncrypt and are quite happy to see NGINX donating to the project.

In this project (MARA), LetsEncrypt support is integrated via [Cert Manager](https://cert-manager.io/) for Kubernetes. This is nice because it supports certs from a variety of issuers like AWS, Google, Vault, Cloudflare, etc in addition to Let's Encrypt.

Re:Nginx Modern Reference Architectures by mholt on Thursday May 19, 2022 @02:07PM (#31438087)
Caddy provides higher memory safety guarantees than C servers.
Re:Nginx Modern Reference Architectures by maxloh on Thursday May 19, 2022 @04:02PM (#31439438)
Garbage-collected language seems to be slower.

Discord have switched away from Go for some performance-critical services because of it. [0] https://discord.com/blog/why-discord-is-switching-from-go-to...

Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @04:51PM (#31439995)
The webserver will rarely be your bottleneck. Your application or database will be long before the webserver is.
Re:Nginx Modern Reference Architectures by maxloh on Friday May 20, 2022 @12:57AM (#31443517)
I don't find it convincing to switch to a slower alternative.
Re:Nginx Modern Reference Architectures by mholt on Friday May 20, 2022 @12:03PM (#31449277)
What are your performance requirements?

Go is fast enough for Google. It's too slow for you?

Re:Nginx Modern Reference Architectures by maxloh on Saturday May 21, 2022 @04:10PM (#31461356)
Aren't proxy server more performance critical than usual backend code?
Re:Nginx Modern Reference Architectures by mholt on Sunday May 22, 2022 @05:53PM (#31473186)
No. The proxy is almost never the bottleneck.
Re:Nginx Modern Reference Architectures by zmxz on Thursday May 19, 2022 @03:25PM (#31438962)
Source? Any content on memory safety concerns in "C" servers or problems that lead to catastrophic results in production? I'm interested in reading about it.
Re:Nginx Modern Reference Architectures by xyzzyz on Thursday May 19, 2022 @04:05PM (#31439486)
Uh, google “heartbleed bug”, that’s just one example of a massive and catastrophic result of lack of memory safety in C. It probably costed something on the order of $1B for remediation efforts globally.
Re:Nginx Modern Reference Architectures by zmxz on Thursday May 19, 2022 @04:27PM (#31439742)
Heartbleed was in openssl. I didn't ask about memory safety in C. Caddy author was pretty accurate with his statement, I asked for proof. Heartbleed is not the only memory leak, there's plenty, C's been around like software written in it. I'm aware of shortcomings.
Re:Nginx Modern Reference Architectures by xyzzyz on Thursday May 19, 2022 @06:40PM (#31441101)
Sorry, I don’t get understand your response. Most Caddy’s competitors use OpenSSL, and so are vulnerable to bugs in it. A lot of those bugs, like for example heartbleed, are only possible due to the nature of the C language. Those kinds of bugs, the memory safety bugs, are prevented when using memory safe languages like Go. These bugs are real and serious, and collectively cost billions of dollars in damages.
Re:Nginx Modern Reference Architectures by zmxz on Friday May 20, 2022 @03:58AM (#31444561)
I don't understand your response.

There's not a single proof except heartbleed in OpenSSL. I merely asked the author to provide the proof, not for you and I to engage in Go's memory safety and billions in damages and inherent nature of C and so on.

If we were to sum all the "damages" caused by faulty software, we'd arrive at a number that exceeds the total sum of money on planet Earth, let's not use that false metric for this discussion.

Is there an actual problem right now with nginx that Caddy circumvents with its architecture? Yes or no? That's the question.

Re:Nginx Modern Reference Architectures by xyzzyz on Saturday May 21, 2022 @01:02AM (#31455372)
Re:Nginx Modern Reference Architectures by mholt on Friday May 20, 2022 @12:04PM (#31449280)
Yes
Re:Nginx Modern Reference Architectures by mholt on Thursday May 19, 2022 @06:34PM (#31441044)
Guess which web servers use(d) openssl...
Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @04:52PM (#31440003)
Re:Nginx Modern Reference Architectures by loganmarchione on Thursday May 19, 2022 @01:35PM (#31437669)
I don't think any large corporations are using Caddy. Everywhere I've worked has used Nginx for any high-traffic applications.

As a side note, I'm looking at Traefik in my homelab and it's honestly difficult because I'm so comfortable with Nginx.

Re:Nginx Modern Reference Architectures by mholt on Thursday May 19, 2022 @02:06PM (#31438072)
There are several large corporations using Caddy. Several companies you've heard of and probably used the products of! I have a call with one of them every few weeks.

I also know a huge hospital chain in the US is using Caddy.

Re:Nginx Modern Reference Architectures by tfigment on Thursday May 19, 2022 @03:25PM (#31438952)
Caddy has problems with streaming gRPC (not simple request/response). So does Traefik to my understanding but Traefik might work better if reports are to be believed. Nginx has support i think but ive not verified it. I like caddy simple config when it works.

None of the proxies seem to do well with bidirectional gRPC streams as they just treat gRPC as a h2 proxy but I'd love to see that proven wrong.

Re:Nginx Modern Reference Architectures by m_sahaf on Thursday May 19, 2022 @04:36PM (#31439845)
Caddy is capable of handling bidirectional gRPC streams! I have just tested it, and it works just fine. Caddy will immediately flush writes when upstream is `h2` or `h2c`[0] instead of having to wait until reading from socket is complete

I used this Caddyfile to proxy to the route_guide example (https://github.com/grpc/grpc-go/tree/master/examples/route_g...):

```

localhost {

    reverse_proxy  h2c://localhost:50051
}

```

It works like a charm.

[0] https://github.com/caddyserver/caddy/blob/e4ce40f8ff04240540...

Re:Nginx Modern Reference Architectures by number6 on Thursday May 19, 2022 @01:55PM (#31437938)
It's difficult because of the boilerplate. Compare traefik with caddy-docker-proxy. Same feature set but only 30% of the code
Re:Nginx Modern Reference Architectures by formerly_proven on Thursday May 19, 2022 @02:37PM (#31438410)
I recall some licensing fuss with Caddy but it looks like it's actually open source now. But stuff like this does tarnish a brand for a long time, see e.g. people still being uncertain about Qt licensing, not because of the more recent developments, but because of licensing issues that were settled in the 90s.
Re:Nginx Modern Reference Architectures by kanonieer on Friday May 20, 2022 @03:37AM (#31444420)
You're absolutely right, it was a dumpster fire and it tarnished the name for me as well.

https://web.archive.org/web/20170913160203/https://caddyserv...

Re:Nginx Modern Reference Architectures by mholt on Thursday May 19, 2022 @06:35PM (#31441052)
Caddy has always been open source. The source code has always been Apache licensed.
Re:Nginx Modern Reference Architectures by eddieroger on Thursday May 19, 2022 @02:07PM (#31438084)
I used to use Caddy in the v1 days, but after some really unfortunately licensing decisions (which I think were reverted ultimately), I felt burned and stopped using it. It was my go-to for static site serving containers for a while, but because some of the time I need to make what they considered "commercial" ones (work related), I couldn't any longer. And it was handled really poorly in my recollection, with some random sponsor header injection. I decided that nginx on a slim distro container is more than fine most of the time.
Re:Nginx Modern Reference Architectures by turtlebits on Thursday May 19, 2022 @01:10PM (#31437301)
Maybe it's just me, but the Caddy config file (either the docs or the format) isn't great. It took me way too long to turn on a simple feature (didn't work, no errors). That plus having to relearn the config going from v1 -> v2 turned me off to it.
Re:Nginx Modern Reference Architectures by megous on Thursday May 19, 2022 @04:44PM (#31439922)
Because there are some benefits to using established software:

    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    E: Unable to locate package caddy
Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @05:28PM (#31440410)
For various reasons, we provide our own apt repo: https://caddyserver.com/docs/install#debian-ubuntu-raspbian

Biggest reason is that debian's packaging requirements are way too much of a burden for us to conform to, especially for Go apps. And they update way too slow for us to be comfortable with.

Re:Nginx Modern Reference Architectures by qbasic_forever on Thursday May 19, 2022 @03:42PM (#31439184)
In the context of kubernetes ingress caddy isn't really there yet, at least compared to nginx (which has two major k8s ingress implementations, one maintained by the core k8s devs themselves).
Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @04:58PM (#31440079)
That is true, unfortunately. We, the core maintainers, don't use k8s ourselves, so we need to defer to the community for help. See https://github.com/caddyserver/ingress
Re:Nginx Modern Reference Architectures by qbasic_forever on Thursday May 19, 2022 @05:53PM (#31440637)
Yeah the whole k8s ingress world is a moving target that has changed substantially even in the last year or two. I don't blame you for not spending a ton of time on it. Caddy is still an awesome option to run inside a pod that needs to have a simple static file server, do some reverse proxying to services in the pod, etc.
Re:Nginx Modern Reference Architectures by chamakits on Thursday May 19, 2022 @01:57PM (#31437964)
From my limited and possibly outdated experience (about a year since I last looked), caddy's documentation was lacking a bit when compared to nginx.

Even worse than that, any time I tried to google to better understand how to do something with caddy, every result returned how to achieve exactly that with nginx; and would rarely find the way to do it with caddy.

But it's been about a year since that, unsure if things have changed.

Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @05:00PM (#31440099)
> caddy's documentation was lacking a bit when compared to nginx

Could you be more specific? We keep hearing vague comments like this about our docs, but without feedback pointing to specific issues, we can't improve them.

I suggest that next time you have trouble finding the information you're looking for, reach out to us and let us know. Open a topic on the forums, we'll be glad to help you find what you're looking for. And if you do so and point out a lack in our docs, we'll know where to focus our efforts.

Re:Nginx Modern Reference Architectures by chamakits on Thursday May 19, 2022 @07:42PM (#31441649)
I don’t remember off the top of my head, apologies. I’ll take a look at my notes see if I can find what I was looking for!

Having said that, I’ll keep this in mind next time I try out again; and if I run into something, I’ll reach out in the forums.

Re:Nginx Modern Reference Architectures by chamakits on Thursday May 19, 2022 @07:48PM (#31441700)
Oh, I’m remembering bits of it!

I was at the time trying to setup jupyter to work with a reverse proxy such that I could access jupyter through zerotier.

Sorry this is all probably a messy set of words that don’t all make sense together with more details. I’ll commit to next time that I’m trying this or anything else, and I hit a wall, I’ll ask.

Re:Nginx Modern Reference Architectures by heurisko on Thursday May 19, 2022 @04:03PM (#31439457)
When I evaluated Caddy it was missing some features, such as rate limiting.

But I think Caddy is a fine server as well.

Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @04:57PM (#31440064)
Re:Nginx Modern Reference Architectures by heurisko on Thursday May 19, 2022 @05:06PM (#31440168)
Yes, I came across that. Didn't really compare to rate limiting in nginx, in terms of maturity or support.

> WORK IN PROGRESS: Please note that this module is still unfinished and may have bugs. Please try it out and file bug reports - thanks!

Re:Nginx Modern Reference Architectures by francislavoie on Thursday May 19, 2022 @05:24PM (#31440353)
Sure, but it'll never reach maturity if you don't try it! We're always looking for more people to test things out, give their thoughts, and contribute ideas or fixes.
Re:Nginx Modern Reference Architectures by POPOSYS on Friday May 20, 2022 @05:26AM (#31445097)
I think that is the problematic point with Caddy right now - if you are using it, you are actually a beta tester. This is a problem for sites with lots of traffic because things start to be different with servers maxed out - Nginx is battle hardened, learned it´s lessons and you can be sure that it works under extreme load as expected.

Also you definitely need rate limiting out of the box, not as some beta plugin that needs to be fiddled in manually.

Also streaming is something that needs really good testing with high loads.

K8s is important for many people, too.

Of course it is not easy to find someone willing to test on a high traffic site when there are well established alternatives since many years and no actual problem needs to be solved.

"Easy configuration" is not very important for admins of high traffic sites. Not easy to find what could be the selling point for Caddy, but it should deliver one or two special things more.

Re:Nginx Modern Reference Architectures by heurisko on Friday May 20, 2022 @02:53AM (#31444162)
I think Caddy should adopt the module as a core module.
Re:Nginx Modern Reference Architectures by francislavoie on Friday May 20, 2022 @02:56AM (#31444184)
We may eventually do that, but we need evidence that it's actually useful for people, since it adds to the long-term maintenance burden if we bundle it.
Re:Nginx Modern Reference Architectures by heurisko on Friday May 20, 2022 @09:32AM (#31447122)
I look through my server logs sometimes and often see connection and request throttling being activated, by those who are hitting the servers with their automated exploit tools. I see having some sort of throttling as essential as being able to set the max client body file size to prevent abuse.

Of course, I could run something else in front of the server, but it's simpler for my use cases not to. The auto-SSL and simplified configuration were very attractive things about Caddy, however.

If your CMS provides it, maybe you could run an exit poll for people who demo Caddy, then look elsewhere, as I'm just speaking from my own experience.

Re:Nginx Modern Reference Architectures by francislavoie on Friday May 20, 2022 @10:27AM (#31447877)
Understood, I'm not saying it's not generally useful, I'm specifically talking about this implementation; is the way it was built useful, does it actually solve user's rate limiting needs? We need people to try to the plugin so we can mould it to user's needs. Then later we can bundle it once we're sure it's designed properly and we're locked-in on the approach taken.
Re:Nginx Modern Reference Architectures by heurisko on Sunday May 22, 2022 @01:31PM (#31470632)
Indeed, I see where you are coming from.

However I wonder if it would work better if the module were brought under the Caddy namespace and marked "beta".

At least then, there would be a show of commitment for the idea, if not the implementation.

Recently I've been looking at the Symfony framework, and this is how they introduce features, with a clear warning that the API etc. is subject to change.

Re:Nginx Modern Reference Architectures by POPOSYS on Friday May 20, 2022 @05:34AM (#31445156)
Rate limiting is a life saving jacket, not some fancy nice-to-have feature.
Re:Nginx Modern Reference Architectures by francislavoie on Friday May 20, 2022 @10:00AM (#31447484)
Understood, but we don't want to ship something half-baked in the standard distribution. So we offer it as a plugin first so it can be worked on and polished without being tied to Caddy's versioning. Please try it out and give specific feedback on the plugin. We need to hear if there's any functionality missing or if the design doesn't fit user's needs before we can bundle it.
Re:Nginx Modern Reference Architectures by kanonieer on Friday May 20, 2022 @03:42AM (#31444452)
I stopped considering using Caddy when the following happened:

https://web.archive.org/web/20170913160203/https://caddyserv...

The project lost tons of good faith back then, and in my eyes (as a brand) never recovered.

Re:Nginx Modern Reference Architectures by c0balt on Friday May 20, 2022 @04:44AM (#31444851)
Idk but for most projects that don't go down the apache foundation or GNU path funding via either support (Tidelift, consulting...) or non-oss extended versions (NGINX has NGINX+) is normal. Passion is nice but money for a sustainable livelihood has got to come from somewhere with donations often being insufficient.
Re:Nginx Modern Reference Architectures by kanonieer on Friday May 20, 2022 @08:35AM (#31446513)
Have you read the article? Because what was proposed back then is not what NGINX is doing.
Re:Nginx Modern Reference Architectures by maxloh on Thursday May 19, 2022 @03:58PM (#31439386)
Candy is written in Go (a garbage-collected language), which in theory slower than C (the language Nginx is written in).
Re:Nginx Modern Reference Architectures by zmxz on Thursday May 19, 2022 @02:20PM (#31438221)
Seeing you're someone capable of using a web server, it implies you're beyond capable of finding why Caddy over nginx and vice versa, which brings up the question: why didn't you?
Re:Nginx Modern Reference Architectures by alskdjflaskjdhf on Thursday May 19, 2022 @03:36PM (#31439105)
This kind of response is rude and unhelpful. This is how they are finding it out. Just googling "X vs Y" invariably gives you worthless SEO spam these days. A forum like HN, where you can hear about the experiences of other people in the field, is a great way of getting more useful information.
Re:Nginx Modern Reference Architectures by zmxz on Friday May 20, 2022 @03:54AM (#31444538)
Thank you for the response and the judgement, but we obviously consider the word "rude" to mean something else.

I was not offensive or ill-mannered, I merely wondered why a person, that's apparently capable of doing minimal research, does not do so before asking for opinion of people on HN in regards to software details. Briefly judging GitHub activity is not the way to make conclusions, which was the only detail being covered.

It's easy to label someone or something rude, I can't control how you read the text I write and what the imaginary tone of voice is. Great Robin Hood display, I love the fact we all get to be judge and jury who decide what's good and what's bad, often missing the real intent.