When a good chunk of those years had interest rates in the 10%+ range, and when the years that were lower than 10% ended with "The Housing Bubble Pop" and "Great Recession", that 6% compounded growth is actually kinda awful.
In any case, compound growth is the center of most economies. Being the owner of things entitles you to profits, be it in stocks, bonds, or real estate.
The "buying on debt" depends on the rate that you got in at. If you're complaining that people in the 1970s were able to get 15% mortgages to earn lol 6% on Real Estate, then I don't think you fully appreciate the history of Stagflation and the 80s bond market.
---------
All in all, housing prices have rose because interest rates have fallen. More people can afford a house when mortgages are at 4% instead of at 15%, and the price of homes go up to compensate.
Unless rates drop dramatically in a short-period of time (like in 2012), refinancing opportunities are rare. From the 1980s, you wouldn't have had a refinance opportunity until the mid 90s.