If you're a guy who's social, cool and fun to hang around with, people will tend to assume that you've slept with more than a few women. If you on the other hand come off as a bitter and angry person, most people will think that your "number" is low or non-existant, because you're displaying some very unattractive traits.
There is a certain type of woman who will look down on you if you have a lower number, namely the status-seeking type of girl. Avoiding this type is probably a good idea regardless of whether your number is bigger or smaller.
No, I think that might be human nature.
I have traveled enough to have observed this distinction, and it's striking. The cultures are very different. In most of the world, nerds are not unpopular; nerds are cool.
The big problem is that American women are prone to "groupthink", and judge a man's desirability based on how attractive they presume other women to find him. So a guy with a low number is perceived as unwanted for that reason alone. Women in the rest of the world are a lot more independent-minded and don't fall to this.
I take it the other way. I've decided that if the majority of women in NYC aren't going to respect me because my number is (voluntarily) low-- not zero, but single-digits-- I shouldn't feel bad about being an unforgiving bastard when it comes to women being too experienced/"stretched out". Regardless of age, I don't think I could marry a woman whose number's higher than 5.