Yes. While diamonds have always been valuable, in the 19th century, DeBeers bought most/all known large mines and exercised their monopoly position to control the market. They also launched massive marketing campaigns touting diamonds as the "best" stone for an engagement ring.

>Or is there some intrinsic reason why the spectral response of the impurities in a carbon lattice actually matters?

It makes a pretty color?

but if people perceive yellow diamonds to be pretty (even if its through marketing), then it is all the same