It took about a year for me to internalize that things will often be that fucked. :)

Conversely, there are 5xx statuses that you shouldn't expect to change with each request, for example 505 (HTTP version not supported) or, really, 500 in some cases.

There's AFAIK no standardized strategy to deal with the different status code. Instead it's important to get the status code right to as accurately as possible describe the error to the client. The server never tells the client what to do; it tells it what went wrong so that the client can deal with the error at its own discretion. In this case the server told he client that it was a teapot when what actually happened was an internal server error. That's wrong because it's wrong, not because every client will or even should deal with either of these errors in a predictable way.

While I was a bit too over-broad in my generalizations of 4xx vs 5xx status codes when it comes to retries, the gist is still the same. The specification does specifically call out when it's okay to retry and when it isn't. And it also specifically calls out when it's okay for intermediate servers and browsers to cache responses and when it isn't. In the case being discussed, 500 is specifically the correct error code because it cannot be cached and can be retried. That's not the case for 409 or 429, which either cannot be retried automatically (409) or can be cached (429). 402 is, IIRC, underspecified and shouldn't be used. 403 specifically states that the request should not be retried.

It's important to make a distinction between the user/application and the user agent when we talk about retries. The user agent is the program or library that implements the HTTP spec. You seem to be talking about the former, who is always capable of retrying a request when they feel circumstances have changed enough for the request to succeed. But user agents are intentionally dumber. They can't resolve conflicts, fix permissions or do anything outside of the logic dictated by the spec. And they have much more limited license to retry 4xx responses or cache 5xx responses.

Today, in similar situations, I'd return a 500 with a message in the body.

My best guess? The request ended up at a teapot and the poor thing was asked to do something that no teapot has ever done before.

Generally speaking I tend to agree, but in this specific case how is sending a 418 "biting you in the behind" worse than sending a more proper 500 error? (Or perhaps a 400? Hard to say exactly, feels like the server is having unexpected errors since it cannot properly parse the hostname.)

So yes, returning a 4XX for a 5XX is problematic.

Since it was a server side error, and the issue was not a client side one, the 500 error code is most appropriate. The fact that the npm client itself is the first one to trigger this error is not the issue here, as it could have just as easily been yarn or another client. Following standards are important, and we weren't trying to make tea.

And error handling is always relevant.

Think about your normal interactions with problem solving as a team. What's more useful, a teammate who tells you when there's a problem what the problem is or a teammate who hides things, obfuscates, and makes random jokes?

400 tells you asked in a way that wasn't understood.

401 says you're unauthorized and 403 says you're not allowed (similar but potentially different implications, not my favorite nuance but it's a common example that might come up)

504 says there's a problem communicating between the two of you.

404 says that your data doesn't exist.

500 says the server is having problems doing the thing and it's completely out of your hands.

418 says "I'm a teapot"

What's more useful data?

Well ideally some 5xx code because it's a server error. I personally think servers should default to 500 if there is an uncaught or unhandled error condition.

That's one simple obvious difference.