The numbers vary between elections and polling organizations (why I dont know) but it seems like ~40-50% of people do not vote so that means that people following your line of thinking are more numerous than you may think. ;)

[1]:https://en.wikipedia.org/wiki/Voter_turnout_in_United_States...

The reason I say numbers vary is because the chart at the top of this article paints a different story (that more than 50% do not vote)