The entire point of returning a status code, at least in theory, would be to not simply be practically-semantically indistinguishable from a 200.

In practice, if you try to return a 402 with an HTML page explaining how to pay, you'll quickly revert that because you'll break some percentage of HTTP clients (not browsers, but other ones) that will try to do "something" with 402, even if it's just assuming "well, it's a 4xx so it's basically a 404", and you'll return a 200 for that page. On the one hand, it isn't necessarily that many clients that will blow up, but on the other hand, you'll observe that no clients will be confused by the 200, so the engineering pressures are pretty consistent here, which is why everybody ends up in the same place.

In theory, if you read the standard, you ought to be able to return a 402, and use content-negotiation to decide whether to return it in a human-readable format, or some rich selection of machine-readable formats, and you ought to be able to annotate it with several dimensions of somewhat overlapping information about exactly how to cache it and exactly what conditions should cause the cache to be expired, including some specification of the HTTP-standard authorization techniques available, etc. etc.... in practice, pretty much every individual element in that collection only sorta works, and trying to do all of them at once essentially impossible at scale, and you'll return a 200 page with HTML and expect the client to pick up the pieces, and if you do offer a JSON view, while you may honor the negotiation stuff you'll probably also allow a querystring or form parameter to ask for JSON too because you can't count on the people writing clients to even know what headers are.... and libraries often make them somewhat less convenient to work with than form parameters, since everybody needs those. And you probably won't "content negotiate" at all, but just have entirely separate URLs for your human stuff and your official "REST API", because most people who try to get clever and make your human website URLs also be the "REST API" pretty quickly discover it's not worth the effort.

I'm sorry, but this is a misunderstanding of the web at a very basic level.

    In theory, if you read the standard, you ought 
    to be able to return a 402 [snip]

Every "problem" you just described applies to 404 as well, and everything handles that just fine.

There is absolutely no problem with servers returning 4xx/5xx codes along with some HTML to render.

All the other things you just described, the content negotiation and so on, are highly debatable at best but the important thing to understand is that they're completely orthogonal to whether or not the server has returned some HTML along with the error code.

You are correct in that human readers don't really know/care whether it's a 4xx/5xx code. But that's fine. They're humans, so it's OK for the HTML returned to give them a choice of what to do.

Non-200 response codes are, of course, already massively useful for other bits of code, for RESTful services and otherwise.

> Every "problem" you just described applies to 404 as well, and everything handles that just fine.

I'm not sure what value of everything you're using, but I remember Internet Explorer's 'friendly error messages'. If your html isn't long enough, and that abomination still exists, you'll get Microsoft's 404 instead of your nice and short 404.

I have worked with many embedded http clients that can't reliable return content unless it's from an HTTP 200. I wouldn't write one like that, but you don't always get to pick your platform, and some platforms give you HTTP or nothing, so you can't use TCP and do a better job (or more often, you have much more important platform issues to work around than to roll your own http when the platform one works enough and it's trivial to just return 200 in case the server had something to return to the client)

> If your html isn't long enough, and that abomination still exists, you'll get Microsoft's 404 instead of your nice and short 404.

The most annoying part is that this knowledge was abused to make even the most basic and otherwise-useless 404 pages show up in IE.

Back when this was a thing, I literally saw this happen. I have one of these pages saved, which is how I know - it was a page which just had "Not Found - The requested document was not found on this server." on it (along with the site name), and then a long HTML comment saying 'Unfortunately, Microsoft has added a clever new "feature" to Internet Explorer.' and explaining the comment's existence.

All for a 404 page that gave far less useful information than IE's own 404 page did. If the page had any brand styling on it I might understand, but this was literally a bare-bones, no-styling page with nothing useful on it.

Also many libraries are built by throwing exceptions for non-200 error codes.

No conflict there.

For SOAP stuff, an exception on 4xx/5xx is generally a sensible design.

For human user-agents (ie, web browsers) there's still no conflict. Deciding whether or not to render some HTML is orthogonal to whether or not some other action might be taken. Generally, you do want to present a choice to the user at this point.

Developers are always doing a if curl status > 400 throw error. I keep running into this pattern in external libruaries. Then they throw the error in place. Please stop this and bubble up the error so the user of the lib can decide.

You do know browsers render 404 error pages, right?

Yes. 200, 301, and 404 are top-tier status codes that even the simplest system can't help but handle in some sensible manner. In the second tier you have the necessary support for partial content and basic content negotiation. After that is when it gets a lot more complicated.

The fact that "browsers render 404 pages" won't save you when you try to get "clever", err, I mean, "standards compliant" and serve a complicated HTTP page out on 402. It probably won't confuse browsers, which will probably just render the page and leave the user oblivious a 402 occurred, which is precisely why I included "(not browsers, but other ones)".

Browsers are forced to march together in some semblance of lock step. When you step out into the heterogeneous world programming language APIs, "convenience" code written on top of those APIs, frameworks written on top of that convenience code, internal libraries written on top of those frameworks... yeah... you can't get very many complicated semantics through that sort of pipeline. And that's not a terribly unrealistic pipeline, unfortunately.

If 402 had actually become part of a means to get paid it would have quickly joined that list of top tier codes right behind 200.

Don't forget these popular code. You were probably 301/302ed at least once today. 201 is popular too. In fact you probably have to do something that will give you a 201 before you get your 402.

301 Moved Permanently 302 Found 400 Bad Request 401 Unauthorized 403 Forbidden 404 Not Found 410 Gone 500 Internal Server Error 501 Not Implemented 503 Service Unavailable 550 Permission denied

What you are saying is true; what I was keeping myself confined to are the codes that you have reasonable assurance that the client will actually do something with. A lot of those codes will not cause the client to do anything terribly specific, like "follow a redirect"; it'll just return the content as if it was a 200, and if it's an API, maybe return the request as failed, but that's about it. In the vast majority of cases, clients don't particularly care if they get 400, 401, 403, 404, or indeed, 4xx.

I wonder if there's even a single client in the world that will get a standards-compliant 412 Precondition Failed and try a second request with a different set of conditions. Or indeed do anything sensibly different at all vs. having received a 404. (I specify "standards-compliant" because I'm sure there's plenty of APIs that use 412 for something non-standard.)

And if browsers did something sensible with I'm A Teapot it would "have quickly joined that list of top tier codes right behind 200". So? We have what we have.

Most modern browsers do.

> The HTTP 418 I'm a teapot client error response code indicates that the server refuses to brew coffee because it is a teapot.

I wonder how FF/Chrome would react to this!

By just rendering it: https://httpstat.us/418

I saw dozens of websites with elaborative 404 pages, like including search form, or even returning site search results with context of requested page, splitting URI to keywords. I see how it is implementable with 402 status as well.

That seems reasonable to me. Browsers ended up dealing with 401 and "Basic Auth", in a semi-standard way, for example. "Payment required" isn't substantially different. More complex for sure, but the context shared seems sufficient.