In theory, if you read the standard, you ought
to be able to return a 402 [snip]
There is absolutely no problem with servers returning 4xx/5xx codes along with some HTML to render.
All the other things you just described, the content negotiation and so on, are highly debatable at best but the important thing to understand is that they're completely orthogonal to whether or not the server has returned some HTML along with the error code.
You are correct in that human readers don't really know/care whether it's a 4xx/5xx code. But that's fine. They're humans, so it's OK for the HTML returned to give them a choice of what to do.
Non-200 response codes are, of course, already massively useful for other bits of code, for RESTful services and otherwise.
I'm not sure what value of everything you're using, but I remember Internet Explorer's 'friendly error messages'. If your html isn't long enough, and that abomination still exists, you'll get Microsoft's 404 instead of your nice and short 404.
I have worked with many embedded http clients that can't reliable return content unless it's from an HTTP 200. I wouldn't write one like that, but you don't always get to pick your platform, and some platforms give you HTTP or nothing, so you can't use TCP and do a better job (or more often, you have much more important platform issues to work around than to roll your own http when the platform one works enough and it's trivial to just return 200 in case the server had something to return to the client)
The most annoying part is that this knowledge was abused to make even the most basic and otherwise-useless 404 pages show up in IE.
Back when this was a thing, I literally saw this happen. I have one of these pages saved, which is how I know - it was a page which just had "Not Found - The requested document was not found on this server." on it (along with the site name), and then a long HTML comment saying 'Unfortunately, Microsoft has added a clever new "feature" to Internet Explorer.' and explaining the comment's existence.
All for a 404 page that gave far less useful information than IE's own 404 page did. If the page had any brand styling on it I might understand, but this was literally a bare-bones, no-styling page with nothing useful on it.
For SOAP stuff, an exception on 4xx/5xx is generally a sensible design.
For human user-agents (ie, web browsers) there's still no conflict. Deciding whether or not to render some HTML is orthogonal to whether or not some other action might be taken. Generally, you do want to present a choice to the user at this point.
The fact that "browsers render 404 pages" won't save you when you try to get "clever", err, I mean, "standards compliant" and serve a complicated HTTP page out on 402. It probably won't confuse browsers, which will probably just render the page and leave the user oblivious a 402 occurred, which is precisely why I included "(not browsers, but other ones)".
Browsers are forced to march together in some semblance of lock step. When you step out into the heterogeneous world programming language APIs, "convenience" code written on top of those APIs, frameworks written on top of that convenience code, internal libraries written on top of those frameworks... yeah... you can't get very many complicated semantics through that sort of pipeline. And that's not a terribly unrealistic pipeline, unfortunately.
301 Moved Permanently 302 Found 400 Bad Request 401 Unauthorized 403 Forbidden 404 Not Found 410 Gone 500 Internal Server Error 501 Not Implemented 503 Service Unavailable 550 Permission denied
I wonder if there's even a single client in the world that will get a standards-compliant 412 Precondition Failed and try a second request with a different set of conditions. Or indeed do anything sensibly different at all vs. having received a 404. (I specify "standards-compliant" because I'm sure there's plenty of APIs that use 412 for something non-standard.)
I wonder how FF/Chrome would react to this!
In practice, if you try to return a 402 with an HTML page explaining how to pay, you'll quickly revert that because you'll break some percentage of HTTP clients (not browsers, but other ones) that will try to do "something" with 402, even if it's just assuming "well, it's a 4xx so it's basically a 404", and you'll return a 200 for that page. On the one hand, it isn't necessarily that many clients that will blow up, but on the other hand, you'll observe that no clients will be confused by the 200, so the engineering pressures are pretty consistent here, which is why everybody ends up in the same place.
In theory, if you read the standard, you ought to be able to return a 402, and use content-negotiation to decide whether to return it in a human-readable format, or some rich selection of machine-readable formats, and you ought to be able to annotate it with several dimensions of somewhat overlapping information about exactly how to cache it and exactly what conditions should cause the cache to be expired, including some specification of the HTTP-standard authorization techniques available, etc. etc.... in practice, pretty much every individual element in that collection only sorta works, and trying to do all of them at once essentially impossible at scale, and you'll return a 200 page with HTML and expect the client to pick up the pieces, and if you do offer a JSON view, while you may honor the negotiation stuff you'll probably also allow a querystring or form parameter to ask for JSON too because you can't count on the people writing clients to even know what headers are.... and libraries often make them somewhat less convenient to work with than form parameters, since everybody needs those. And you probably won't "content negotiate" at all, but just have entirely separate URLs for your human stuff and your official "REST API", because most people who try to get clever and make your human website URLs also be the "REST API" pretty quickly discover it's not worth the effort.