You could likely find something anti-science about most people. For instance a mind boggling number of people believe in ghosts, should they be disinvited? With regards to a security conference, it’s no more harmful of a world view than believing the Earth is flat. What about anti-vaxxers? Ban them too? Maybe come up with a list of beliefs you think are ideologically pure enough for the conference?