Compare it to the business world, where if you ship a pallet of widgets and a few of them get damaged, you can't just tell UPS "you broke my shit, I'm not paying you." No, they delivered your shit, however subpar, so they expect payment for the trouble. If you keep withholding based on principle you'll find yourself blacklisted and have your account sent to collections.
And to that: a server is very rarely personally responsible for a bad experience. Restaurant service is a team effort and the service you get merely reflects how well management functions.
Also while I can't speak to why other people don't tip, the reasons why I (very occasionally) don't are usually under the server's control. Even if the immediate cause of my unhappiness was not under their control, they had the option of acknowledging the problem and offering an apology. It takes a lot to lose my tip.
But if I take my clothing to a drycleaner, and my clothing comes back torn due to their incompetence, I'm not expected to pay for the drycleaning. Ditto if the valet damages my car.
The fact that someone put time in does not entitle them to my money. If I receive something of value, I give something of value. If this makes me a person that you do not wish to know, then I'm happy that I don't know you because you sound like an unpleasant person for me to be around.