And while there is definitely no hope for binary blob consumer software to ever have real end-to-end encryption, there is hope that it could happen for open source software, distribution of which is not controlled by software vendors with many competing parties that package and ship it and an incentive not to sneak anything in because of that.
Note that the client side apps are available for researchers to study. If they find a back door, it will be a big publicity issue for the app vendor.
Your concern is still valid though. Both Whatsapp and Signal could announce tomorrow that they drop E2EE or add some feature (like a default on non-disable-able cloud based spellchecker) and there is nothing you could do to protect your correspondence.
Matrix is different here that it encourages independent implementations as well as end to end encryption.
I bet this is also why the intelligence agencies aren't pounding on them for having E2E encryption, obviously Google is an open book to them. And Google has access to all your history for their own purposes.
Of course you can choose to turn off the backups, but you can't be sure all your contacts are doing the same.