As you can't verify their claims, all you can do is trust. If you want end-to-end encryption you should be gpg encrypting every mail, not relying on the unverifiable word on a provider. Any provider worth its salt will know that, thus wouldn't actually advertise an insecure (as verify can't be done) system as a secure one.
- https://protonmail.com/blog/protonmail-open-source-crytograp...
- https://protonmail.com/blog/protonmail-secure-email-open-sou...
You don't have to trust any of their server code - you only have to trust that the JavaScript blob they send you is actually the same as the open source version. This is the same threat model as trusting Signal from the App store instead of side-loading it yourself.
I feel like the hate is a case of people thinking not being perfect is worse than being average or bad.