OK, just to cite one example: The population of North Carolina as of 2012 was 9,752,073. 23.7% are under 18 and therefore ineligible to vote. That leaves approximately 7,440,831 eligible voters[1] (I'm ignoring convicted felons and other edge cases for now).
According to the State Board of Elections, there are currently 6,438,531 registered voters. 2,764,322 Democrats, 1,984,251 Republicans, 20,901 Libertarians and 1,669,057 unaffiliated.[2] So that gives us around 1,002,300 potential voters who aren't even registered. So around 13% of the eligible voters aren't even registered and obviously don't vote.
Now that's just one state, and that doesn't even attempt to account for people who are registered, but don't bother to vote, or vote infrequently. But it does suggest that there's a not-insignificant number of people who are sitting on the sidelines during elections.
Edit: Also, as one might expect, there's a Wikipedia page on this topic:
http://en.wikipedia.org/wiki/Voter_turnout_in_the_United_Sta...
http://felonvoting.procon.org/view.resource.php?resourceID=0...
Your North Carolina example still stands, as only (only being relative here) ~80,000 people are disenfranchised due to felony status. But, in states like Florida, Mississippi and Kentucky, it can be as high as 10% of the voting age population.
I bet most of them are people who don't care about politics. Good luck trying to engage them, people have been trying to do so for decades and have failed. The simple reality is that life is comfortable enough for most. Few are starving, few are persecuted- if you're earning enough to live a relatively comfortable life, why bother voting?
The political equilibrium keeps things as they are. Those in power can't do what needs doing because they'll get thrown out for it (ex.: total spending must be cut in half, but nobody is going to make such draconian across-the-board cuts and survive). Something has to break to disrupt that equilibrium, something like defaulting on debt maintenance or onset of hyperinflation.
Yes, we're stuck in a rut. There comes a point where you have to get out of the car and start walking, leaving the driving paradigm behind.