All things degrade due to entropy. Things cannot truly become more valuable by themselves.
I think raw land is one thing that can become valuable on its own. Of course billions of years from now it will be gone.
Where I live the land is very much a big part of the cost of the home. The structure is worth what it would cost to demolish plus rebuild it minus existing wear and tear/needed repairs I figure. The value of the structure doesn’t appreciate as fast as the land it is on due to degradations as you’ve pointed out and materials/labor usually not increasing as fast as appreciating assets which the land itself is. Of course in many locations the structure is worth more than the land it is on.
Hence location being the most important part of the equation as any real estate agent will tell you.
The land doesn’t become “valuable on its own”, it’s just used to extract value from other things.
> In that original state of things, which precedes both the appropriation of land and the accumulation of stock, the whole produce of labour belongs to the labourer. He has neither landlord nor master to share with him.
> ...
> As soon as land becomes private property, the landlord demands a share of almost all the produce which the labourer can either raise, or collect from it. His rent makes the first deduction from the produce of the labour which is employed upon land.
Wealth of Nations
Another nice thing about owning a home is you can improve it (additions, adding bathrooms, etc) and certain additions more or less pay for themselves in terms of "getting your money back" over enough time. Again, probably not going to outperform the market, but you get to live in it. So even if the market tanks for a few years, I still get the benefits of whatever I've invested in my home.
The trick is to not overspend on a home. Too many people use all their savings to make a downpayment and then use too much of their income to pay for it. This can not only get you in trouble but also puts you at a serious disadvantage in terms of accumulating wealth.