Yes, you would. Whether speech is racist or not is a characteristic of the speech, not determined by the speaker’s skin colour.