The elegant balance of the system is that everyone eligible to vote in Athens were eligible to be selected for office (through random lot). Officials weren't some distant elite separate from the voters, it could be your neighbor, it could be you next.

This sort of creates an incentive to moderate the use of ostracism to the times someone really fucked up bad.