I always thought it was related to older terms that linked hate with fear, like "yellow peril". Older racist beliefs like this often played into fear, and I can't help but wonder if historical hatred of LGBT+ people was also based on fear.