I'd argue that the fact that they're rare, special things found in the earth is a big part of the story. At least from people I've talked to who prefer earth grown diamonds, this is something that's come up multiple times.

However, people always turn a blind eye to the part of the story that involves cruelty, which is largely down to De Beers excellent marketing and creation of artificial scarcity.