IANAL, but the difference is that in this case, QI came into being as a (very narrow) reading of federal law (§1983), which congress can amend easily. The Loving case was decided on an interpretation of the constitution, which as you say, is much harder to amend.