At least one axis here, alongside the gay-straight axis, is dominance behavior. One can feel dismissal or dislike towards people whose behavior registers as submissive. There are varying degrees to which gay men manifest this -- and considering whether your anti-homo feelings correlate with it can be informative -- but many definitely do.