Until now, Apple hasn't offered any way of actually comparing what keys you are writing to, though. In other words, there was nothing technically stopping Apple (or somebody with the power to compel them) from adding additional recipient keys to your account that your or your contacts would not be aware of.

This is about to change with contact key verification and transparency, though.

Asymmetric encryption doesn’t change anything. Apple is still the one telling your client whose key to encrypt the data for.

A BIG part of successful encryption is key exchange.

A successful SSL/TLS man-in-the-middle attack exchanges false asymmetric keys.