See, though, this is the thing. It's great that you're not being all "lol male tears", have a cookie, but you're putting a lot more effort into defending your conception of feminism, and into drawing a distinction between yourself and "lol male tears" and making sure we all know you're not like that, than into addressing the topic at hand. That's what is getting you downvoted, not some imaginary knee-jerk reaction to a word.
It's not accepted to derail a women-centered discussion with "but patriarchy hurts men too!"; why do you expect anything less in derailing a men-centered discussion with "but feminism is for men too!"? Granted you're not being That Guy in the usual direction. But you're no less being That Guy for it. Maybe consider not being That Guy.
But some people don't want to see a reasonable counterbalance. They see a word they don't like and downvote without reading; they actually get angry when others fail to live down to their stereotypes. Going in the other direction, I've seen liberal feminists get confused and angry when they meet a conservative who supports GLBT rights. It makes people uncomfortable when folks don't stay in their pigeonholes. It's frustrating.