You can't please everybody, which leaves your conscience as the deciding factor. If you would rather not draw black people with googly eyes and vacant expressions, I can hardly criticize that. Like you point out, you'll get criticized - but again, it's impossible to make everyone satisfied with this stuff.