It's probably more that in the grand scheme of things trans-men don't pose any sort of threat to other males. I don't recall ever discussing trans-men with my peers.
Trans women are seen as seeking female privileges (like the ability to compete against women in sports, which is easier than competing against men). Which enrages some people.
Trans men are seen as giving up their female privileges, to which everyone shrugs and says "you want less privilege? Fine by me!"
I can think of at least one -- working in tech.
Of course men have privileges too, but I think many of men's privileges come from confounding variables. Like men are on average taller, and people are biased to view taller people as having more authority, so taller people are more likely to become CEOs etc.
Whereas women's privileges come from society compensating for men's privileges. So society sees women on average are smaller and physically weaker, and thus decides they need privileges like separate sports to compensate.
Trans women who went through male puberty often have the size advantage of men, so them also have the social privileges of women is seen as double dipping.
While for trans men it's the opposite, they're often small like women, but not afforded social protection for it.
Anyway that's just my theory why some people hate trans women more than trans men. I myself have sympathy for both cases, dysphoria sounds pretty awful.
Transwomen, on the other hand, are much more defined, in the eyes of a patriarchal society, by their rejection of masculinity in favor of femininity. To some cis-men, they appear as aberrations or duplicitous (hence the nickname "trap"), to some cis-women they appear as potential unfalsifiable unknowns, and a potential thing to be feared for sexual violence. Transwomen are thus caught in the crossfires of fear from both genders.