The problem comes when the terms spread enough that miscommunication happens. When someone says "Bob is a neo-nazi" meaning "right wing", but the person they are talking to hears "wants to kill jews and blacks and gays". And then Bob gets lynched/ostracized/fired.

When the meaning of a word isn't likely to have a big impact on people's lives we can be a bit more forgiving (though personally I still want to move towards a clearer thinking population).