I don't believe I have the relationship flipped.

There are many valuable things that are not made into jewelry. There are metals that are more valuable than gold, but they are not as commonly used in jewelry because they aren't as aesthetically appealing. Gold was used for making beautiful things in cultures that did not treat it as money. It became valued because it was uniquely workable, beautiful, and durable.

Part of gold's value is its rarity, certainly. But another part of it is utility. If investor demand for gold collapsed, it would still be valued.