Its oddly worse than you say. Most crypto code I have seen is of below median code quality. It seems to be written by people with neither a strong academic crypto background nor a solid practical coding background.

Compare e.g. OpenSSL, the most widely used library, with SQLite, the most widely used embedded database. OpenSSL is a mess under the hood with a hideous API on the surface, while SQLite is beautiful code with an API that is a joy to use.

Newer trendier stuff like libsodium are better but they still feel unpolished to me.