It looks like the top 11 delegates control around 40% of the vote[1], so nearly effective control. That does seem like a small group, but if a delegate was acting maliciously, I'd expect a number of their holders to change delegation.

[1] https://sybil.org/#/delegates/ens