Hence "depending on what is implied". You make a remark because something surprises you, and the reasons for that surprise might be benign or they might be racist.