True, but we do accumulate wealth, in the form of houses, infrastructure and other quite permantent stuff.

We do, but our rate of nominal wealth accumulation bears no relation to the rate at which we accumulate tangible physical assets. Not to mention that most of those physical assets depreciate, which must be accounted for.

I know many people that save gold and silver

Gold and silver have very little inherent value, only trade value (other than a few industrial applications, but they can often be replaced by copper). We don't save very much food, or surplus housing, or surplus machinery or petroleum or solar panels, mostly because these things degrade physically and become obsolete when saved. Instead, we rely on today's economy to provide these things.

Total worldwide gold production amounts to $132 billion, or only 0.2% of world GDP. Even if we saved all the gold mined every year, it wouldn't support any substantial amount of real saving.

Well we do keep some commodity reserves, but these don't show up on anyone's account books as "savings". They show up as illiquid assets.