I should have clarified that I was primarily referring to the stuff dealing with all the wrapper formats (like PKIX certificate verification), not the core cryptographic algorithms themselves.

The core cryptographic algorithms, IMHO, should be written in a dedicated language for writing cryptographic algorithms so that they can get formally-verified constant-time assembly out of it without having to complain to us compiler writers that we keep figuring out how to deobfuscate their branches.