It goes both ways. In the early days of colonization of the Americas, Spain suffered inflation due to an influx of silver.

The point is, the amount of money needs to correlate with the size of the economy.

There's no single physical good that naturally has the required correlation, so using physical goods as the basis of money is foolish.