Exactly my point. Many hold the philosophy that discriminating between the two in any context is an attack.

Most heterosexual men identify as being attracted to women, and don't hold trans women in that category.