Not always. As soon as hate becomes widespread enough walking away besomes impossible. That's why things like Holocaust denial are so dangerous is being left unchecked.