This is a massive and common misunderstanding. Publishing a block is NOT voting on its validity. Every node, regardless of stake accrued 'votes' (the correct word is 'chooses') independently of the validity of blocks just like in Bitcoin.

Blockchains that allow protocol changes via votes are an option, and how you quantify votes is a parameter, but PoS is not tied to being a government token, and 'votes' on forks (its really not a vote as both protocols exist after a fork) are made by large and small holders alike. It has nothing to do with amount of currency owned.

Mining isn't exactly a fair shake either... at least in PoS more network participants are even able to "vote".

Staking better aligns incentives when compared to mining and most other alternatives.

Yes, with POS aren't we just back to a (now programatically enforced!) cabal of bankers that control the money system?

Nope, it's still distributed consensus rather than centralized. If one banker stops being nice, he'll loose all his funds. The current system either ignores people not-being-nice or ejects them, but they'll never lose what they made.

Compared to PoS where the biggest incentive is to be nice.

The fear is that there will be a powerful in-group that has majority control, not just a single bad actor.

Correct me if I’m wrong, but isn’t that a problem shared with PoW?

I would argue that POW tends towards specialization of the hardware, congregating power in few hands, so yes. Yes it is.

This, I find, to be the biggest flaw with crypto in general.