I've had many occasions where someone has said something that caught me a little and offended me and I ultimately just decided they didn't mean it that way (even if they did). I do this because I know I'm not the smoothest individual out there and I have no doubt I've said some things that may have causes someone else doubt or offense when it was just me saying things in a bad manner.
When you enter the work force there's a certain amount of bullshit you have to put up with. I once had the CEO of a company roll his eyes at me in the middle of a meeting and say roughly "you IT people are all the same" in front of everyone there.
Sometimes it's just a part of being it the work force and you have to deal with bullshit in your day to day. It doesn't mean it's sexist, racist, or whathaveyou, it just means you're dealing with people who are oblivious or assholes.
Because I think that usually, they are innocent. But that means I also probably miss out on (and don't have to stress out about) some sexism that occurs. And that helps me 1) not burn out like you mentioned, and 2) not be disliked by work by bringing it up and making a (potentially completely valid) fuss. So it probably helps me, oddly.
I can't say that how I evaluate these kinds of things is correct, or even good, but I do think it's probably helped my career more than it's hurt.