OpenSSL ciphers are generally good, and there's no need to rewrite that.
Certificate and protocol are where most of the tricky bugs would be found.
Protocol is actually not that much code, and a from scratch rewrite that only did TLS 1.2 and 1.3, and only allowed for currently deemed reasonable options wouldn't be too bad to do. Of course, that would eliminate renegotiation, so today's OpenSSL bug wouldn't be possible. I've done a TLS 1.3 client from scratch and it's about a monthish of prototyping, followed by six months to do it right and test and deploy and find bugs and have it audited by a 3rd party and eventually be happy with it. I did the prototyping, another developer did the production code; we did have other developers familiar with TLS to review and assist. I imagine TLS 1.2 would be similar, but having experience from 1.3 would help it along. Server side wouldn't add much, once the session is established everything is the same, it's just having to flip parsing and generating of the handshake pieces.
Certificate/x.509 handling is a beast. I imagine rewriting and testing that would take a long time. You would probably find bugs in other verifiers while you're doing it if you cross test though. You could save a little by limiting to commercially available certificate options and limiting verification modes, but it's still a lot. ASN.1 parsing might be significantly nicer not in C though.
I concur with everything you wrote.
Re: ASN.1, you really want a compiler for it, not hand-rolled codecs -- I don't care how good your library and macros for hand-rolling ASN.1 DER codecs might be, it's going to yield very difficult to maintain code full of bugs.
ASN.1 is not the biggest part. Certificate validation, revocation status checking, and everything to do with that, and all the many certificate extensions and subject alternative name forms, and... all of that requires a very large body of code, and there's no formal specifications for any of it.
Better options for standard data structures and initialization: https://github.com/openssl/openssl/commit/fb9fa6b51defd48157...
Error handling: https://github.com/openssl/openssl/commit/2a40b7bc7b94dd7de8...
That said, of course you can end up with a similar bug in Rust, there are just nice ways the language encourages practices that would avoid it.
This would be based off using the #[no_mangle] extensively, I presume, probably with some amount of "unsafe" usage? At which point has the primary use case of Rust in this situation been lost?
(note: not commenting on if one should or should not use it, only mentioning it's existence)
https://github.com/ctz/rustls/blob/master/audit/TLS-01-repor...
A lot of software was written a long time ago with as-of-yet-unestablished good practices.
Almost no concern for security or anything else.
I wonder if they actually took really good best practices both for security and just software in general, used something like Rust, used a very open dev process with a lot of eyes on it - and somehow avoided the feature creep / political feature orientation trap ... if it could not only be re-written, but made more simple.
Maybe it would take 5 years (1 design 1 develop 3 deploy and test) but it would still be worth it.
I believe that security is a much worse problem than it needs to be, because everything we depend on today was designed without much security in mind.
Our current OSs for example, were designed in an era where security was no concern. Any app can do 'anything'.
If Linux, Mac and Windows were to have been designed from the start with good containerization, no direct memory pointers (unless special case), if we were using ATM-like networking with identity etc. ... the world would be an entirely different place.
It's obviously unlikely to happen, but if the powers that be actually wanted it to happen, it would.
I would hope that the US military/DARPA designs a system that is hardened in a way that could be made use of in the civilian sector as well. Though it's obviously very unlikely.
Especially with such a sensitive thing like crypto. Just off the top - you have to be really careful with implementation to prevent timing attacks. And that’s just one of many hundreds things you need to care about.