You still trust the server, unless the encryption is done with code that wasn't delivered from the server. E2E prevents your content from being stolen in a data breach or from being accessed if the server was fine when you sent a message but compromised later.

Good point. The (variously named) security code should allow you to withdraw even that trust (assuming you verify the security code and the binary on your client...), right. Or does it? If the server knows the secret, it can invisibly MITM you, right?

Unless you're independently verifying the keys in meatspace you're still trusting the server.