The most annoying part is that this knowledge was abused to make even the most basic and otherwise-useless 404 pages show up in IE.
Back when this was a thing, I literally saw this happen. I have one of these pages saved, which is how I know - it was a page which just had "Not Found - The requested document was not found on this server." on it (along with the site name), and then a long HTML comment saying 'Unfortunately, Microsoft has added a clever new "feature" to Internet Explorer.' and explaining the comment's existence.
All for a 404 page that gave far less useful information than IE's own 404 page did. If the page had any brand styling on it I might understand, but this was literally a bare-bones, no-styling page with nothing useful on it.
For SOAP stuff, an exception on 4xx/5xx is generally a sensible design.
For human user-agents (ie, web browsers) there's still no conflict. Deciding whether or not to render some HTML is orthogonal to whether or not some other action might be taken. Generally, you do want to present a choice to the user at this point.
I'm not sure what value of everything you're using, but I remember Internet Explorer's 'friendly error messages'. If your html isn't long enough, and that abomination still exists, you'll get Microsoft's 404 instead of your nice and short 404.
I have worked with many embedded http clients that can't reliable return content unless it's from an HTTP 200. I wouldn't write one like that, but you don't always get to pick your platform, and some platforms give you HTTP or nothing, so you can't use TCP and do a better job (or more often, you have much more important platform issues to work around than to roll your own http when the platform one works enough and it's trivial to just return 200 in case the server had something to return to the client)