I'm not saying we should ignore them. Just that for most people the two concepts "sex" and "gender" aren't separate, and trans people are simply regarded as quirky. For example there are people born with 6 functional fingers per hand but we still say that humans have 5 fingers per hand and don't say that humans have anywhere between 0 and 8 fingers per hand.

Treating sex and gender separately makes it seem like they just accidentally coincide in case of 99.x% of the people.