Why is that unacceptable to say to women, but acceptable to say to men? When people say that to women they call them sexist, but the same people say that to men themselves, so obviously there is sexism here.

I think if you accept that for men, you should accept it for women, and vice versa.