desselc
Your assumption seems to be that if no-one physically stops these men, they can and will do what they like, with impunity, and that there's no chance of them simply respecting women's boundaries. Is that what you mean?
Thankfully, this fad is on the way out, and with it, the overvigilance that has led to unfortunate misunderstandings like you mention in your comment - none of which are justification for males to impose themselves on female spaces.
I would be very surprised if colleges letting males enjoy women's locker rooms, and therefore not providing women any single-sex place to change, isn't ruled as being sex discrimination.
Keep in mind this issue is about privacy and dignity and safety for women, not about giving men whatever they want whenever they want.