Honestly, I think the only real solution is to sue these accusers for defamation. If there's actual evidence that you are racist, then you can just take your lumps and don't respond to them.

However, if you want to accuse someone of racism, you better have evidence of you're going to pay.