Until now, Apple hasn't offered any way of actually comparing what keys you are writing to, though. In other words, there was nothing technically stopping Apple (or somebody with the power to compel them) from adding additional recipient keys to your account that your or your contacts would not be aware of.

This is about to change with contact key verification and transparency, though.