TL;DR: Restrictions on free speech are primarily used against the powerless, not the powerful. So don't create free speech restrictions that you think are going to be used to protect blacks, say, against white racism. Those restrictions will be used primarily against blacks, not against whites.