You say that "some people who identify as the same religion [as me] have been homophobic" but I wonder if that's not minimizing the truth. If you're part of an opt-in demographic that's known for negative characteristic X, even if you yourself don't exhibit that characteristic, you're going to receive some transitive association -- and it's at least somewhat warranted because your participation is voluntary.