It depends on what aspect of men and women you mean. Socialization, biology, endocrinology, reproduction? Saying "Men and women are different" is an easily defendable conclusion. However, the implications that you are making from that regarding anatomy, physiology, and psychology are very hard to prove. There is significant overlap between men and women in each aspect.

PS: This type of argument is called a Motte and Bailey. You provide two arguments, one weak and one strong then fall back to the strong argument when challenged.

https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy