The big problem is that American women are prone to "groupthink", and judge a man's desirability based on how attractive they presume other women to find him. So a guy with a low number is perceived as unwanted for that reason alone. Women in the rest of the world are a lot more independent-minded and don't fall to this.
I take it the other way. I've decided that if the majority of women in NYC aren't going to respect me because my number is (voluntarily) low-- not zero, but single-digits-- I shouldn't feel bad about being an unforgiving bastard when it comes to women being too experienced/"stretched out". Regardless of age, I don't think I could marry a woman whose number's higher than 5.
If you're a guy who's social, cool and fun to hang around with, people will tend to assume that you've slept with more than a few women. If you on the other hand come off as a bitter and angry person, most people will think that your "number" is low or non-existant, because you're displaying some very unattractive traits.
There is a certain type of woman who will look down on you if you have a lower number, namely the status-seeking type of girl. Avoiding this type is probably a good idea regardless of whether your number is bigger or smaller.
No, I think that might be human nature.
I have traveled enough to have observed this distinction, and it's striking. The cultures are very different. In most of the world, nerds are not unpopular; nerds are cool.
During past bad relationships, or phases when I was "racking up a number", my work was definitely impaired.
How did I prevent it? I suppose I didn't, but I mostly just worried about being a decent human being to people I know, and not sweating whether casual sweating was immoral.
Side note: once you get to about 30, I don't think anyone talks in terms of your number, and your ability to find a "quality woman" (itself an error-prone reduction of many dimensions to one) will not be changed by even a few score of scores. I sense that your philosophy may lean towards the Christian from your finding consensual casual sex immoral, so your fitness function for quality woman might be way different from mine, and the women who rate highly over that function might actually care about your "number". IMO, if you're still single at 30, no one ought to shocked that your number is well into double digits.