The value placed on a good is measured by what a person is willing to give up in exchange. Naturally only those things one actually has available to give up count—not what one thinks one might be willing to pay in a hypothetical world where affordability is not a concern.

The millionaire just has a longer history of already having given things up to amass that much money which they can draw on in exchange for goods later. (Or, in the case of gifts/inheritance, others gave things up on their behalf and for their benefit—which amounts to the same thing.) That's exactly what money represents: more value provided to others than the owners have claimed for themselves.