You are right in your fear.

I don't know if you have already linked it yourself, but this is why there has been orchestrated removal/damage of confederate monuments in multiple cities recently. They are trying to taint and delegitimize the original writers of the constitution by associating them with slave ownership/racism, in order to then delegitimize their product, i.e. the constitution and especially the 1st and 2nd amendment rights.