Yes - it's legal to discriminate by sex. If you go out looking for apartments in a major city, you will see many "women-only" options, probably for some of the reasons you listed. However, I don't think the arguments for sexual prejudice apply to racial prejudice.