We, the people who aren't interested in unicorns, are rarely mentioned on lists of fantasy fandom although we do share the same feelings about a particular element of the fantasy genre.

I don't see why some see the need to clump people together by what they don't believe in. Atheism isn't about "interest" or "feelings", so your comparison doesn't hold either. Different people arrive at the same conclusions via completely different paths, many not even rationally. It's really arrogant to assume atheists are superior in this regard (and I'm not saying you said that but it's not too hard to infer), and I say that as an atheist myself.