There are great historical precedents. The first democracy, Athenian democracy, allowed only 50,000 out of 300,000 citizens to vote. https://en.wikipedia.org/wiki/Athenian_democracy