Cryptography is uniquely difficult to audit precisely because it is encrypted. It requires more effort to tell what an application is doing if all its input/output are completely incompressible.

Maybe cryptography should be handled exclusively by the OS. Software would communicate through a protocol that can be easily audited (like normal http with special headers). Firewalls could be used to verify what is going on and normal tools could be used to inspect the data. That kind of checking is essential and doesn't require the talent needed for code review.