Question then: who decides whether someone is "rightly" upset? There is some implication there that we don't need act on every offense taken.

For instance, I don't expect others to act (i.e. should be forced to change) because something offends me.

Notes:

Don't confuse this question with me arguing that we shouldn't be empathetic.

I'm also not from the U.S. so my approach and offenses are different culturally than your own.