Give an example. I must admit I am at a loss for words. What should I feel guilty for, and to whose benefit?
Everybody faces prejudice. For example I face prejudice because I work with computers. I don't think that warrants inflicting a huge guilt trip on the rest of the world.
Sure, you never see it... that's makes it easy to pretend it's not happening. But anyone with eyes below your nose level (most likely women) definitely sees it.
And your only defense is developing good rapport with those people, and giving them reason to believe that you'll respond graciously when they point it out.