>It does only affect them though, but that is not the same,especially when people are using this argument (as you do) to insinuate that people opposing gay marriage think gay people are lesser beings or something like that.

Actually to the best of my knowledge, it is the same, at least under US law. De facto discrimination is still considered discrimination.