If you are in a relatively high position of power/OK with potentially losing your job, publicly calling someone out, even your boss, for something like this is risky but the right thing to do.

Even small stuff like calling BS on the sort of comments that happen when only guys are around (we've all heard them).

The implicit consent of everyone around these kinds of people is what lets this stuff continue (along with everything else of course).