Regarding toilets, it is because women's toilets have also traditionally served as a 'safe space' (i.e. without men). It is somewhere they can go without being subject to (for example) what is colloquially termed the 'male gaze'. Hence the disquiet over trans-women using such spaces, and the lack of disquiet over the reverse.

Of course in a fully enlightened society arguably women should not need such safe spaces, but I'm not sure we are quite there yet.