If fewer voters is better than 0 voters is optimal!
A nice illustration is that most open source projects that are based on democracy are always torn apart by inner power plays but those that have a 'benevolent dictator' seem to flourish.
I'm quite uncertain how useful it is to observe systems like open source projects where the stakes are so very low compared to governing a polity.
The extreme of dictatorship says nothing about the middle, such as a republic with a limited franchise (e.g. own land or pay a hefty poll tax), or adding other types of bodies to government. Such as the US Senate back when its members were picked by state legislatures, or a power sharing arrangement between a monarch and a legislative body or three, like the U.K. of long ago.
It's definitely not the best exponent of what I was trying to get across, but I didn't say all that much initially. At least when they're not fighting wars, or are degenerate cases like the Saudi royal "family" (15,000 members, 2,000 with wealth and power per Wikipedia), there tend to be limits to their rapaciousness. Much less so with traditional oligarchies.