You might not like it, but every programming language in existence has easy to use HTTP and JSON parsing libraries. It's basically a solved problem.
As a software developer I dread working with protocols that are not via HTTP because most often then not the clients are badly designed, leaky and I can't fix it by changing the client.
And I love JSON in spite of its inefficiencies, because it is easy to debug and there are really good JSON parsing libraries for the statically typed languages I'm using (Scala, Haskell). You basically declare your high level types and the library will derive (de)serialization and validation logic for you.
Also consider that this isn't necessarily about building full fledged email clients. Maybe you want an automated way to read an Inbox and import certain messages in your database. Maybe you build a browser extension that counts the number of unread messages. There are many, many micro tasks that developers would be interested in doing, if they had an easy way to do it.
And do you know what protocol does NOT have good libraries for every language in existence? IMAP.
It's basically not a competition from my perspective, HTTP/JSON wins by default.
The nice thing about the old internet protocols is that they tried to distinguish between abstraction and implementation layers. (Not always successfully or effectively, but they did at least try.)
Modern protocol designs seem to do the opposite. They start from a default implementation layer - usually REST, HTTP, and JSON - and then try to build the abstraction layer out of that.
This isn't necessarily a good approach. But it is a reflection of the difference in culture - the change from hackers who were comfortable with open-ended bottom-up development, and modern developers who seem to prefer snap-together development using standard libraries.
Also known as the Unix philosophy ;-)
This isn't about comfort, but about doing what needs to be done.
Every webmail interface in existence has implemented their own half baked, proprietary protocol that does what JMAP is doing. Or do you think Google's Gmail web interface is using IMAP?
---
> "hackers who were comfortable with open-ended bottom-up development"
That's because they lacked basic tools. Hackers in those days weren't getting much work done either.
Wat.
I was one of those hackers ;-)
That argument doesn't hold much water. Every programming language in existence also has easy to use socket libraries. Are you implying that HTTP and JSON are less error-prone than whatever-over-TCP? There's nothing magical about either HTTP or JSON that makes them the right tool for every job. But I guess when all you have is a hammer ...
The tooling is better; debugging is easier; the interfaces are less finnicky; the transfer is less brittle; dealing with objects and documents is much easier conceptually than streams; very powerful performance optimizations and catching can just be turned on while being transparent to the app; crappy networking equipment won't ruin your day; corporate firewalls won't get in your way; encryption is basically drop-in with transparent tooling support.
In no way is HTTP/JSON the best protocol for every situation but thinking you can do better with something home grown on top of TCP is ignoring all the human effort that's gone into solving the problems with straight-tcp protocols. So yes any protocol that's had as much love put into it would probably be just as good but few things have.
We did actually extent IMAP with some experimental stuff first (cross folder threading and thread collapsing for example) but it never solved the batching and proxying difficulties and the cost of instantiating message numbers per-session that are un-avoidable parts of the IMAP model. By the way, engineers who work on other servers have said similar things to us - IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability, etc.
Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design, it assumes a centralized control plane (think Raft or Paxos), i.e. a single central server with many distributed clients, which is a shame for new systems building on CRDTs.
How does JMAP address this achilles heel of IMAP? Merkle tree sync?
More detail on the problem: https://news.ycombinator.com/item?id=20479011
I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago...
https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July...
Thanks for this. It was a good read and promoting UIDs instead of bumping UIDVALIDITY is a great idea.
Surely, it would be even better if IMAP had support for distributed Merkle Tree synchronization, so that clients can sync with different server replicas without the server replicas having to coordinate or promote UIDs at all in the first place?
I don't know of any algorithm other than a Merkle Tree that will do this correctly and simply and efficiently in a distributed setting?
https://datatracker.ietf.org/doc/draft-ietf-extra-imap4rev2/
Merkle tree would is great for full history - I built something like that with digital signing for tracking clinical data over 15 years ago that would have been fine for email as well - but it's way over-expensive for email.
Which leads back to the key design principle I try to use when working on protocols, which is something like "make the weird edge cases sand and possible, but don't optimise them!" You don't want to build the Merkle tree complexity into a client and have all the servers be completely independent, because it makes the protocol a pain to use for the simple case.
(What we use for IMAP replication in Cyrus is much dumber than all this - it uses a per-message CRC and XORs them together for the whole mailbox to create the sync_crc value. If those don't match, it does a resync. Fixes up split brain just fine while being cheap and dumb for the common case.
Does JMAP then not support syncing full history efficiently as a key design priority?
> That would be a pretty massive change to the entire data model of IMAP. It's never going to happen[tm].
Sure, and that's why I had hoped JMAP would, especially in a distributed master setting, since this was something IMAP overlooked.
I guess it's because JMAP still needs to be backwards-compatible with the strict consistency master-slave restriction of IMAP?
> What we use for IMAP replication in Cyrus is much dumber than all this - it uses a per-message CRC and XORs them together for the whole mailbox to create the sync_crc value. If those don't match, it does a resync. Fixes up split brain just fine while being cheap and dumb for the common case.
You're implementing the first layer of an incremental Merkle tree right there. How many bits are those CRCs just out of interest?
The thing JMAP doesn't have, that I think you're trying to ask for, is the ability to talk to divergent endpoints and resolve the state on the client. JMAP expects to talk to a server which is merging the state into a linear world for it. That linear world view can be different from different servers.
For example, if you implemented your server as a series of git commits, complete with merges whenever you'd had changes on two different servers, then you could just use a commit hash as the state string - and if you got an oldState which was a commit that didn't happen on your server you could still use git to calculate the series of changes on your branch since the fork point plus changes on the remote branch after that commit - and you'd be able to issue a full set of changes. If you DON'T have at least that commit and its predecessors locally, you can't calculate the diff, so you have to return cannotCalculateChanges.
I think what you're wanting here is for the server to get from the client what the last fork point was, and give the client just the changes it knows about since that fork point (like a git fetch) and then the client calculates the merged changes. That's not how JMAP is designed to work. It's an interesting idea, but it does lead to very high client complexity, so it's not what we wanted JMAP to be.
The CRCs are 32 bits, but they're not the only signal. There's also modseq and uidnext, so the CRC doesn't need to be cryptographically strong - if somebody deliberately creates a replica state that comes up with the same CRC but has different content then they could make servers believe they are in sync, but in that attack vector they probably also have enough control to just make the replication layer lie.
We're also using sha1 for message content integrity, which I'd like to move away from eventually. There are plans, but we have other things to finish first. We do insert some randomness into a header on message delivery which makes it much harder for someone to calculate what will be injected when sending somebody an email via SMTP!
No, not "history" in the git sense, just "history" in the simpler email sense, i.e. "email history", all past emails in their final converged state.
How does it do this in a distributed setting? When clients might be connecting to multiple servers with different states?
I mean, you can do other things - but at the point where you're switching between multiple masters in a split brain from your client, all bets are off - so you're going to need to fall back resyncing any mutable data.
CRDTs handle split brain fine. They eventually converge. You just don't want your clients flapping in the meantime by resyncing tons of data. That's where a distributed sync algorithm would help keep things working in the interim.
Just use NNTP! /s
:)
HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al.
But you can't send mail to anyone using gmail, which is nearly everyone, unless you play by Google's rules. And even then they'll still put you in their spam folder, or just randomly blackhole you without any explanation or recourse.
JMAP vs IMAP vs POP is moot as far as I'm concerned. As long as my mail client continues to work, I'm good. If this gains any traction, I'm sure dovecot and the likes will add support for it (if they haven't already). But no amount of protocol fiddling can fix the fundamental issues of email centralization.
That is why protocol fiddling is an essential part of it. Not enough, but without something better than IMAP might as well give up yesterday (as most already have done).
How's that? I don't see how messing with IMAP/POP/JMAP will make a difference there. Mail delivery, over SMTP, is what the Gmails and Outlooks of this world are so nasty about. They claim to protect their users from spam, but small mail servers get killed in the crossfire. Mail clients downloading mail over whatever protocol is a trivial problem by comparison, and they have zilch to do with SMTP abuse.
If 99.9% of all mail originate from gmail+outlook then they can (and will) pretty much ignore the rest.
The less dominant they are the more they are forced to be reasonable.
JMAP will help diversify the market as well as simplify the process of creating competing products. And has the potential to vastly increase demand for native clients and proper support from webmail vendors.
All of the above will work towards lessen the dominance of gmail etc.
It isn't anything by itself but I believe it is a requirement going forward.
Because IMAP doesn't cut it, so everyone has to create their own crap, which won't be as "good/easy/cheap" as gmail.
How would JMAP accomplish that, in a way that other protocols don't?
Another reason is that webmail vendors of today typically deliver a subpar IMAP/POP experience. So combining a native client and webmail when on the go is rarely ideal.
JMAP is the first chance in a long time to get a decent experience that doesn't involve an expert and (even if you are willing to put in the time) tons of compromises.
And aside from all that we have an opportunity and reason to revive native clients that has gone stagnant (which is pretty much all of them).
Could you elaborate on this? I personally find using a mail client like Thunderbird to access my mail account far more responsive and seamless compared to using the webmail interface. If I don't have much bandwidth to work with, the local mail client using IMAP and SMTP will work while the webmail client will experience timeouts.
But syncing mail with IMAP isn't good enough and many features of webmail clients do not map to IMAP clients. There are also tons of caveats during configuration so tons of people, rightfully so, don't bother.
Those are the premises for JMAP, from which the rest follows.
JSON does not cope well with binary data, which is common in emails. It fails to elegantly deal with the various email encodings which exist in the wild. Consider as well that all JSON numbers are floating point numbers, despite the fact that floating point numbers provide absolutely nothing of value to this spec and in fact are more likely to introduce bugs than not. And embedded devices can't deal with floats quickly or elegantly, but still need to implement them if they want to use JSON-based protocols. And for what gain!
In short, JMAP reinvents the wheel but worse for the sake of making it easier for web developers to build web shit around email. This kind of stupid change for change's sake is an epidemic in the webshit scene. IMAP is warty but it's fine. Let it be. When your only hammer is JavaScript, everyone else's thumb looks like a nail.
IMAP really isn’t fine. If it was, mail providers wouldn’t have kept on making their own protocols/APIs for their mobile apps or their webmail. JMAP has various concrete benefits over IMAP. JMAP has been made by people that have been working with IMAP for decades, and have been actively involved in IMAP improvements over that time as well.
Pretty much all of your objections seem to me to either be misinformed, or addressed at https://jmap.io/. Strongly relevant are headings 2, 3, 4, 5, 6 and 8. (I could write more about individual issues, but I’ll just leave it at that.)
The main thing that is only kind-of addressed there is your objection to JSON number representation. JMAP does not state that JSON is the best thing out there; but rather that it is good enough. JMAP doesn’t use floating-point numbers, uses opaque strings for IDs (where many APIs people home-bake would use numbers), and expressly limits its integers to the −2⁵³+1 to 2⁵³−1 range where it does actually use integers. (And none of those places should ever be anywhere near that.)
I must be missing something obvious -- how is IMAP a push protocol? All experiences with IMAP I've had are in the form of basic request-response flows.
Also, regarding JSON floats, most languages have the ability to give you errors if you encounter a float in a JSON field that is meant to be an integer. So it really shouldn't matter at all that JSON's floats are awful. I haven't looked, but does JMAP use floats?
I don't particularly love HTTP nor JSON, but if that's the only problem with JMAP then it's such a massive improvement over IMAP that we should have started switching to it yesterday.
As Microsoft does it with RPC is just a mess, they do a http call with 2billion bytes in request size, let the connection terminate at timeout and fire up a new call. I for one hope not that this is how JMAP solves it :(
I'm sure that as a practical matter, it will be a rare implementation that does not work by deserializing non-integer decimal numbers into floating point numbers, but the JMAP spec does not require this behavior, and I believe it uses no decimals.
- Email protocols are all shit. The email ecosystem still didn't evolve into a format where we can safely transit binary data around. It is still risky to rely even on octet encodings, so there is very little binary data on emails anyway, it's mostly 7bit strings.
- The numbers in emails are nearly all strings. The very few integers are encoding content length and are not necessary with HTTP.
- We have much better odds of having message push that actually works everywhere with an HTTP protocol, mostly because many of our computers are configured to serve ads first, and ads use HTTP push.
That said, I do look forward into all the interesting problems HTTP will create for large emails.
The only significant characters in email are carriage return, line feed, and period. There's also a line length limit in the SMTP protocol specification. Other than that, bytes sent during the DATA phase are sent unaltered.
Base64 encoding is meant to address this, but it results in a 33% overhead in attachment size. On the usenet side, people came up with an encoding scheme called yenc that actually only escapes those characters mentioned above and only as a 2 to 3% overhead over the original file size.
JMAP seems to not use any float, so implementers do not need floating point logic and can just reject JSON with floats.
Aren't most email attachments base64 encoded anyway? If so, JSON not supporting binary is not really a problem.
It's the default for certain larger email hosts - that's the reason you see it frequently, but that in no way means that it is the _best_ option for a protocol that suggests it's solving the problems of the old one.
By no means is IMAP a particularly good protocol, but it is extremely broadly supported and designed more with email in mind than JMAP appears to be, which seems to focus more on bringing shiny web tech to email regardless of its utility in such a context. It's also one of many closely interlinked mail-related specifications, and JMAP fails to integrate well with many related specs imo.