I agree that there is a middle ground - when someone's quality of life is seriously suffering because, for example, someone is shouting abuse at them all day then something must be done. However, taking serious offence at people's ignorant opinions or ways of expression is unwise and in itself ignorant.
I'm anticipating prejudice so I shall address it pre-emptively - I am, in fact, a minority. In many places on this planet, I risk serious injury just for being what I am. People have expressed extremely ignorant opinions about the group I fall under in workplaces. I initially took offence upon hearing what I heard but then I moved past that stage and accepted that one cannot understand something they haven't dealt with directly. They were not trying to be mean, they merely did not understand because of their limited experience. How can I take offence when this is the case?
That's certainly a good sign that social sanctions for sexism, racism, etc work. People ~should~ think before they speak, much more than they do in most sites and blogs.
Suppose through a sequence of unfortunate events you find yourself needing to walk through a bad section of town. You have your choice of two roads. Down one road you see a group of 5 young men of a race that commits crimes significantly higher than average. Down the other road there's a group of 5 middle-aged women of a race that commits crimes less than average. Is it acceptable to be racist, sexist, and ageist in this scenario?
There's an old saying which appears to have been lost somewhere along the way:
"Offense can only be taken"
Live by it and you will never be offended. It is not the job of everybody around you to constantly walk on eggshells just to keep you happy. You are not at the center of the universe, after all.