Do you see this system of the elite ruling as an ideal that will never be reached (a la Plato's Republic) or as a practical system? Because in the latter case, you've got an intractable problem on your hands.

The problem is a basic political systems one: at some point, you need the consent of the governed. There are no stable systems on the record where a majority of those governed dislike the leadership. Either the oppressed are a minority (colonial Europe, slavery in the US) or the gov'ts are short-lived and unstable (on an order of 50-100 years, in the short term these can work). The most stable regimes that don't have majority support are ones that suppress free speech, which is hard to argue is operating "in everybody's best interest".

So, if you want a stable oligarchy that preserves some free speech, you need a group of "elites" that the whole society agrees is fit to lead them. I don't think the average voter believes themselves to be in the best position to run the country -- that's exactly why they vote for somebody else to do it.

But if you take voting out of the equation, you're trying to build a system where you have an elite that has support of the majority, without consulting the majority. At best, you have a system of guesswork that results in something similar to democracy (because the oligarchy are people the majority supports anyway). At worst, you have an unstable government.

It's a good point. But are there always only two groups, the elite and the majority, or could we create more? It seems to me power works best for everyone when it is more distributed, less uniform, and exists on a gradient from the elites down to the majority. Maybe a government structure could enforce that. For example, instead of just America's, Executive, Legislative and Judicial branches, we might also have additional branches for Education, Finance, Security, Business, etc., each having some (but not always equal) constitutional power over other branches, and all trickling down in power to the majority that has voting rights.

What you're saying is basically boils down to maximizing the complexity (integrated information) of the system. That is possible by increasing differentiation and integration. It is better power to be divided in more fragments, each differentiated from the rest, and then integrated with each other so as to be forced to create a dynamic equilibrium. That's what our brains are doing too, on a scale much more complex.

As reference, the concept of integrated information was developed by Giulio Tononi but I applied it in a different field.

Exit rights are a good starting point for a solution.