You are right if we go older. But we don't have to go far to prove the point.
I could barely afford a house built as a hunting bungalow in 1918, expanded multiple times, in the 60s-70s a lot of unpermitted work was done. Trying to get work done was a nightmare because nothing was to code (some reputable contractors refused to do work). Sure, the appliances were fairly new, the pipes and some of the electrical had been updated, but all of those were relatively minor expenses and most of the upgrades were aesthetic.
Your insurance splits out the cost of the land and "improvements" (the structures) because you're only insuring the house because a flood/fire won't destroy the land. In my case, the cost of rebuilding the house dwarfed the cost of the property itself.
The cost of improvements on land has decreased.
Maybe a 4x price increase is indeed excessive, but building a house like it was possible 50+ years ago, would be just illegal.