They could lose control through a substantial shareholder lawsuit. A minority shareholder suit against Musk could prove tremendous damages repaid in stock and the judge might not let the special classes of shares with extra voting power protect his control.

It would have to be a very good cause though, disliking him or ordinary bad business decisions won't cut it.