The -phobia suffix doesn't indicate fear exclusively, it's also used to describe profound distaste and dislike. Just like with homophobia, One doesn't have to be specifically afraid of gay people to be homophobic, though that is often the case.