And nice strawman, by the way. I have had some EXTREMELY popular blog posts that I bookmarked (remember NVIE’s post on git branching?) 404 just because the site changed architecture. (Now hosted on GitHub.) It’s all laziness and lack of forethought and planning. What is agile about one of the most popular blog posts in the development world 404ing?
That said, yes, it can be a bit time-consuming to map old URLs to new. I once started a project that would help considerably with this, but my employer at the time decided it wasn’t worth it (hey, would they see any more gold coins if they helped clients’ visitors’ bookmarks and search results work?). But a manual connection isn’t impossible. In some cases, it’s easy. When I moved off Drupal (damn it to hell), I redirected all those old /node/<id> URLs to their new ones. Because, damn it, I wasn’t going to add any 404s to the Internet if I could. I didn’t blog to make money. I blogged to help people. And maintaining URLs helps people.
It's the UX inertia of having to write mod_rewrite rules that ensure its never a priority and thus never happens.
That is your mistake. The URI should be decoupled from the technology you are using. The URI is part of the content and not of the software behind the site.
If your stack requires tight coupling with URL structure, you have doomed yourself already.
Nope, it's the lack of a crystal ball. The technology world moves fast. And breaks things, as Zuck says. I have no idea how a site of mine might be structured even a year from now, and if the currently URL scheme will conflict directly with its needs or not, or if an old-to-new translation will be trivial or overly resource-intensive. By now, the world has mostly realized that over-planning is bad, and agile is important.
So who cares if a 5-year-old URL to a page virtually nobody ever visits anymore doesn't work. It's easy to Google the keywords in the link, and you'll probably be able to find the content if it's still around.
(Obviously big sites have an incentive to keep their links working, but they don't need an article from the W3C to tell them that.)