> Why is that unacceptable to say to women, but acceptable to say to men?

Because this is typically in response to allegations of rape? I haven't really noticed any stigma associated with this sentiment outside of that context.