There are a lot of broken incentives. If one of the elite comes to the realization that they are no longer the most useful person for the job, can we trust that they'll step down? Or will they hide this fact from everyone else and continue to enjoy their position of power?
These types of problems are not solved and it's important to keep looking for better solutions. I just don't think that "all people have equal say" is a better solution.
The problem is a basic political systems one: at some point, you need the consent of the governed. There are no stable systems on the record where a majority of those governed dislike the leadership. Either the oppressed are a minority (colonial Europe, slavery in the US) or the gov'ts are short-lived and unstable (on an order of 50-100 years, in the short term these can work). The most stable regimes that don't have majority support are ones that suppress free speech, which is hard to argue is operating "in everybody's best interest".
So, if you want a stable oligarchy that preserves some free speech, you need a group of "elites" that the whole society agrees is fit to lead them. I don't think the average voter believes themselves to be in the best position to run the country -- that's exactly why they vote for somebody else to do it.
But if you take voting out of the equation, you're trying to build a system where you have an elite that has support of the majority, without consulting the majority. At best, you have a system of guesswork that results in something similar to democracy (because the oligarchy are people the majority supports anyway). At worst, you have an unstable government.
As reference, the concept of integrated information was developed by Giulio Tononi but I applied it in a different field.
The usual trouble with this sort of idea is who gets to decide the "elite" that makes the decisions. How do you prevent this group from becoming an insular clique and debar other people from similar opportunities etc.
Power, imho, has a tendency to be self-propogating; people who achieve it hold on to it far after it is useful and tend to bestow it on their proteges, irrespective of their suitability to the task etc. (Anyway it is an opinion; you may disagree)