"People are all privileged in different ways and in different circumstances" is pretty closely tied to the concept of intersectionality.
And yes, there are specific situations where generally unprivileged groups have privilege over generally privileged groups. But there are two main reasons nobody wants to focus on those situations:
1. Those situations are typically so minor and so rare that addressing them just isn't a priority,
2. The same prejudice that causes those situations is generally far more problematic in the opposite direction.
You might want to read this, if you haven't: http://goodmenproject.com/featured-content/what-about-the-me... . It's aimed at a male audience that is basically unfamiliar with privilege, so anyone who's well read on the subject is likely to cringe at some of their oversimplifications. What's interesting about it is that it uses the point that men are hurt by sexism as a call for more men to get involved with feminism.
That's not evidence against there being a systemic imbalance. It's further proof of it.
I never said nor intended to imply there wasn't. But there's imbalance in every system too. Which now makes it sound like I'm saying "Just shut up," which is really why I tend to avoid discussions like this because they become No Win. For everyone. Gah.
Anyway, I think that the fact that systems have imbalance is why we need to talk frequently about what imbalance exists, and how we might fix it. It's also why we should stop this irritating habit of turning everything into a system of processes and functions that don't allow for human error. Plenty of people in the tech community think that a series of rules is the end-all be-all cure to every problem, and refuse to admit that rules often cause as many problems as they solve. Learning to listen and empathize with other people is a more valuable trait than ever, because there are more ways than ever to get away with not doing it.
People used to be taught that. They need to be again.