It's an unfortunate quirk of how the internet has developed that email has stuck around, like postal mail, as long as you put the correct address on the outside of the envelope, you can put almost anything inside.
RFC 822 makes it clear this is a feature:
> In this context, messages are viewed as having an envelope and contents. The envelope contains whatever information is needed to accomplish transmission and delivery. The contents compose the object to be delivered to the recipient. This standard applies only to the format and some of the semantics of message contents. It contains no specification of the information in the envelope.
The format / protocol do impact and limit what the client can do. In a funny way, thanks to MIME, you can technically do whatever you want with emails, but that will be a poorly organized and structured whatever. In many way, emails are like HTML : A way to present information, not to structure and/or organize it. And therefore, there is only so much the client can figure out about the email. Thanks to Thread-id and the recipient list, you can group emails more easily and more relevantly, but that is about it.
Otherwise, yes, email message format is a mess. I also had a pleasure of writing a MIME mail message parser in C. But it was quite fun to try to do it in streaming fashion with no dynamic memory allocation allowed.
My take way is I have no idea how on earth people have actual working IMAP implementations.
I'm not sure why you say emails are hard to search. Extract the text and put it into an engine like lucene. There are the usual small corpus problems, but these aren't dependent on format.
As threads are fundamentally a graph, so complaining that the structure matches the structure of the data is... Odd.. to me.
Including the time zone in the date is a small wart. Add an extra specifier to the format string and move on. Base64 is similar: a bit annoying, but decode and move on. And use RFC1652.
Traditional email threading was based on the contents of the In-Reply-To and References headers. The algorithm is described in RFC 5256 for the IMAP SORT and THREAD extensions[1] and used in several mail user agents[2].
>RFC 68388
what are these numbers, I don't recall RFCs going into 5 digits yet
This is definitely talking from someone who never had to write software that deal with emails. The reality with emails ? They are an horrible format. There is several RFC describing the "email" format, the usual reference is rfc2822, but technically, you can find other rfc that relate to it (ex, the MIME rfcs: RFC 20452, RFC 20463, RFC 20474, RFC 20485, RFC 2049, RFC 68388, RFC 4289, ...). There is a lot that left to be desired in those RFC in term of specifications. They tend to gloss over some part and also allow several format for retro-compatibility.
But then, even the part that are clear and well defined are not respected. I wrote software that dealt with emails for a company that receive several thousands emails on several mailbox everyday. The number of non-complying email that we received that broke our implementation was staggering. Nobody respect the date format, that is standard any more. Then we had some weird stuff, like fields that were mime-encoded even though the spec explicitly say it must not be, attachment metadata that were written completely wrong, headers that made no sense, ... And this was not from your small email provider, it was emails that were coming from a gmail mailbox, outlook, yahoo, ...
Oh and MIME is the best footgun in existence. It allows you to do anything and so of course, you receive anything, including mail-in-mail-in-attachement-in-mixed-content-in-multipart. And you better hope that the boundary are not bugged.
I wrote HTTP servers, IRC clients, BMP parser, and many more format. Nothing ever came close to the mess that is email. Anything is better than email, that is not the clients fault, it's the format fault. We just keep using it because of network effect. But you can bet that any other communication protocol / format sucks less than email.