There are crimes I can be accused of that will enhance my reputation rather than diminish it. There are perfectly legal things I could be accused of that would destroy my career and my social relationships. If your argument is about reputational damage, then using criminality as the threshold for "It's not ok to make a public accusation" is entirely arbitrary.