Because you don't vote people out, you vote people in (or abstain from voting, which is just letting someone else decide who to vote in).

If all the options on the ballot are scoundrels, which seems to often be the case, you're kinda stuck.