I'd see an oligarchy as more of a site like Reddit or News.YC, but rather than every person having a vote, a set group of people are able to. Like if only the leaderboard could perform certain tasks.
The problem to circumvent then would be if the leaderboard became overtly political, but I think that a) that isn't going to happen on a site like this, and b) it should be easy to make the system balanced if we have Dictator Paul Graham around to make sure that the people in such a system don't turn into absolute jackasses.
The problem lies when normal people stop thinking that they can submit and get content to the top, the submissions tend to become rote, and you lose community interest.
(As an aside some of the power users on Digg started taking bribes and were banned en-masse IIRC, but I don't see that happening here.)
It won't happen, but I think it would be an interesting experiment.