It's kind of like "socialist" - many people think of it as inherently bad, instead of just a different set of relative values. I've lived in Sweden and can say that atheism, agnosticism, and socialism aren't bad, just different.