Probably a similar thing as with the word "shrill" — it's a word that's mostly applied only to women, so some people interpret the word itself as being sexist (apparently notwithstanding the fact that women have higher voices).