That is basically the definition of hardness. It doesn't make gems indestructible; they can be worn down like anything else. We cut diamonds even though we don't have something harder to cut them with; it's just much harder than cutting softer stones with diamonds.

A cut gem's sharp corners will be worn down over time by the paper the gem is kept in. Paper's not that hard.

(And of course, even if scratching were completely impossible, gems are still easy to smash. There are all kinds of deformations that aren't scratching.)

I've had a very nice kitchen knife get stuck in an unexpected bone. The blade lost a big chip, because it twisted in my hand after sinking in a little. But no, the bone didn't scratch the blade.

You can easily smash a diamond with a hammer, even though the diamond is far harder than a hammer.

Aside from that, metals quite often gouge and chip one another, and 100+ year old swords are not automatically stronger or made of better materials than present day pot metal.

After a certain point, a sword is more valuable and useful as an art piece than as a weapon outside of emergencies.