News for hackers. Stuff that matters.
Re:Ten Years of JMAP by gjmveloso on Monday December 23, 2024 @04:50AM (#42493172)
JMAP is the best email protocol out there that nobody uses (besides Fastmail), sadly.

It would be great that the two big email providers (Google and Microsoft) implemented and supported it.

It would make so easier and reliable to have a single client that works really well across personal and business email accounts, for example.

Re:Ten Years of JMAP by mort96 on Monday December 23, 2024 @04:55AM (#42493201)
Sadly, Google and Microsoft are in the business of getting their users to use their own gmail and outlook clients. They don't want to be invisible IMAP/JMAP back-ends, they want to own your e-mail experience.
Re:Ten Years of JMAP by CuriousSkeptic on Monday December 23, 2024 @05:03AM (#42493240)
Besides, MS already have their own solution that goes way beyond just mail: Microsoft Graph
Re:Ten Years of JMAP by Mailtemi on Monday December 23, 2024 @05:09AM (#42493279)
MSGraph is slower and not as convenient for email compared to JMAP. In addition to requiring more API calls for the same result, MSGraph starts throttling with HTTP 429 responses after several calls, making the initial synchronization much slower than JMAP or even IMAP.

Hopefully, next year there will be JMAP support for Contacts. As RFC drafts, there are already JMAP specifications for Files and Calendar...

Re:Ten Years of JMAP by CuriousSkeptic on Monday December 23, 2024 @05:25AM (#42493344)
No doubt JMAP is superior for the problem it solves. I wonder if it will matter in the end though. I have the impression MS is positioning it self to be the only viable AI platform where mail and calendars are just a tiny part of that. So while not really competing in the same domain it may be that both IMAP and JMAP may feel as relevant to the world as IRC or XMPP in the not too distant future.
Re:Ten Years of JMAP by mort96 on Monday December 23, 2024 @05:47AM (#42493452)
I am more than happy to not use an "AI platform" for my e-mail and calendar!
Re:Ten Years of JMAP by pas on Monday December 23, 2024 @06:32AM (#42493657)
Sure, and we're the vocal (?) minority here. The SOHO/corporate/family world will uncritically use whatever shit MS regurgitates for them for all the usual reasons.
Re:Ten Years of JMAP by stonogo on Monday December 23, 2024 @05:17AM (#42493319)
And they standardized MTA-STS to ensure that you must use web technologies if you want to use email.
Re:Ten Years of JMAP by dgeiser13 on Monday December 23, 2024 @12:22PM (#42495965)
Gmail supports IMAP.
Re:Ten Years of JMAP by Mailtemi on Monday December 23, 2024 @02:03PM (#42496777)
It does behind CASA!
Re:Ten Years of JMAP by mort96 on Wednesday December 25, 2024 @06:33PM (#42511966)
It does, but IMAP is clearly not a priority, and they certainly aren't interested in JMAP. They support IMAP (with caveats) because you can't really offer e-mail without IMAP, but they want you to use their GMail app.
Re:Ten Years of JMAP by retrochameleon on Thursday December 26, 2024 @02:17PM (#42517095)
Normally, email providers let you organize into folders. And IMAP (if I understand correctly) DOES NOT allow clients to see those folders and replicate the folder structure.

But Gmail doesn't have "folders". They have "labels". I wont go into all the details about how Gmail labels work that frustrate and annoy me, but the point is that IMAP clients cannot see labels, they can't replicate it on the client end, etc.

This caused a legitimate problem at work when the finance dept. wants to use this service that reads emails to automatically scan invoices into their system and whatnot. They wanted to be able to only have a subfolder automatically imported into the program, because too many of the emails were not relevant to get imported, creating more work to clear them out.

We were not able to achieve a solution with a simple folder for the users to move emails into to get them imported. Instead, we had to set up a second email account specifically for this importing service, and whenever finance wants something imported, they forward the email to that second account.

Re:Ten Years of JMAP by MaKey on Thursday December 26, 2024 @04:38PM (#42518034)
> Normally, email providers let you organize into folders. And IMAP (if I understand correctly) DOES NOT allow clients to see those folders and replicate the folder structure.

I think the "NOT" slipped in there by accident.

Re:Ten Years of JMAP by retrochameleon on Saturday December 28, 2024 @01:06PM (#42532977)
Yes it did. Rereading and reediting snafu.
Re:Ten Years of JMAP by amaccuish on Thursday December 26, 2024 @07:25PM (#42518960)
The problem is that in IMAP the concept of labels doesn't exist. Thus they are implemented using folders, which is a concept in IMAP. But that issue is, a message is then in multiple folders (to reflect that it has multiple labels), but in the way this happens in IMAP, they are different messages, even if the content is the same. And thus I can imagine, automated programs, not aware of this, may for example see 2 invoices (because the email has 2 labels) rather than 1.
Re:Ten Years of JMAP by k_g_b_ on Friday December 27, 2024 @03:38AM (#42520748)
IMAP has a concept of flags/keywords (https://datatracker.ietf.org/doc/html/rfc9051 2.3.2) and there's a whole bunch of servers and clients that support using it in a similar way as labels - Thunderbird, Zimbra, dovecot, etc.

It's not a feature originally built for labelling like Gmail, so it has a few complexities ("Servers MAY permit the client to define new keywords") and enough clients (especially mobile) that don't support it - but they could just like Thunderbird or some webmail clients.

I've used it basically exactly like Gmail labels for years on TB on desktop.

Re:Ten Years of JMAP by joecool1029 on Tuesday December 24, 2024 @07:56PM (#42506099)
What about Amazon? They could do it with AWS. It seems odd now to think they haven't offered a service like this yet.
Re:Ten Years of JMAP by hyhconito on Monday December 23, 2024 @05:09AM (#42493274)
I think the first "big tech" adopter would be Apple if anything. Their entire email stack is standard protocols.

But you have to ask on that scale, what would it bring them other than change if everything already works fine?

Re:Ten Years of JMAP by Mailtemi on Monday December 23, 2024 @05:41AM (#42493422)
JMAP uses fewer resources server-side, and with their scale, it will probably reduce operating costs. But it will depends on how much they will save compared with devel/migration costs.
Re:Ten Years of JMAP by hyhconito on Monday December 23, 2024 @08:18AM (#42494180)
It may use less sockets but that doesn't necessarily mean less resources. Parsing JSON is expensive, especially when there are binary types interleaved in it.
Re:Ten Years of JMAP by Mailtemi on Monday December 23, 2024 @09:31AM (#42494627)
Can you give an example of expensive binary data parsing?

Having implemented JMAP and IMAP protocols myself, I haven’t encountered a need in either protocol to send binary data to the server for sync/search operations that would require the server to perform expensive parsing.

JMAP offers many improvements, such as a stable messageId for each message and a state mechanism that allows the server to be queried for changes since the last saved state. This avoids the need for numerous IMAP SELECT commands per folder to check the state using CONSTORE/QRESYNC.

If CONSTORE/QRESYNC aren’t supported by the client, it results in very costly chunked queries just to verify if message flags are still the same.

The same applies to SEARCH—if a user has many folders, it requires multiple network hops to SELECT and query each folder. With JMAP, this can be done in a single API call.

Re:Ten Years of JMAP by jeltz on Monday December 23, 2024 @09:22AM (#42494552)
Nah, JMAP is optimized to make fewer and cheaper calls to the server. I agree that it was a mistake to use JSON for JMAP but I doubt the overhead of using JSON matters in practice compare to the benefits. Parsibg JSON with SIMD instructions is very cheap.
Re:Ten Years of JMAP by noprocrasted on Monday December 23, 2024 @05:29AM (#42493368)
> what would it bring them other than change if everything already works fine?

IMAP is absolutely terrible on unreliable connections. A connectionless protocol based on HTTP would do much better in those conditions.

Re:Ten Years of JMAP by hyhconito on Monday December 23, 2024 @05:46AM (#42493443)
I'm not sure it does if I'm honest. The encapsulation via JSON is quite bad and transactional model of HTTP is non-existent. One of our key problems on our platform is deciding on how to handle failures like that in transactional systems. Email is to some degree transactional due to the distributed state. It's going to get ugly.

IMAP is ugly but that is all well understood.

Re:Ten Years of JMAP by brongondwana on Monday December 23, 2024 @06:54AM (#42493765)
The big benefit with JMAP is the state strings and /changes operation which allows even in the face of failures to recovery quite cleanly. It's really more a data synchronising protocol (RFC8620) with email support on top (RFC8621) and the new protocols coming in.
Re:Ten Years of JMAP by hyhconito on Monday December 23, 2024 @08:17AM (#42494168)
Section 3.10 of RFC8620 scares me. I know a lot about that and a lot of the complexity around that trite point is missing.

Note I've designed and built a complex messaging system that runs over HTTP over unreliable connections and run it in production for 15 years so I know what can and does go wrong.

Popcorn at the ready.

Re:Ten Years of JMAP by Semaphor on Monday December 23, 2024 @09:24AM (#42494566)
It’s not as if Fastmail haven’t run JMAP in production at a pretty big level for quite some time by now.
Re:Ten Years of JMAP by hyhconito on Monday December 23, 2024 @10:43AM (#42495178)
And I dumped them because their client was buggy, doesn’t work offline on mobile devices and I had to use IMAP anyway…
Re:Ten Years of JMAP by spondyl on Monday December 23, 2024 @04:12PM (#42497663)
> doesn’t work offline on mobile devices

To some degree, that isn't true anymore as of last week but of course, you meant when you used it, not generally: https://www.fastmail.com/blog/offline-in-beta/

Re:Ten Years of JMAP by hyhconito on Monday December 23, 2024 @05:52PM (#42498246)
Oh so some of it works offline.

Useless to me. Seriously.

I write a lot of email on planes and do a lot of email management.

All that works on the default Apple stuff as shipped other than account management. Using IMAP and iCal etc.

Re:Ten Years of JMAP by NoahKAndrews on Monday December 23, 2024 @07:27PM (#42498680)
Have you looked at their list of what doesn't work offline? It seems like it's mainly attachment management limitations than limitations of managing the emails themselves.
Re:Ten Years of JMAP by throw0101b on Monday December 23, 2024 @07:23AM (#42493890)
> It would make so easier and reliable to have a single client that works really well across personal and business email accounts, for example.

Does not Thunderbird work with Gmail/Microsoft/etc? Does not Apple's Mail.app?

Re:Ten Years of JMAP by jeltz on Monday December 23, 2024 @10:25AM (#42495043)
Not well. At least Thunderbird with Gmail does not work very well.
Re:Ten Years of JMAP by Snacklive on Monday December 23, 2024 @11:17AM (#42495479)
What are the problems you have faced ? I'm been using Thunderbird for a few months and i haven't notice anything weird
Re:Ten Years of JMAP by BenjiWiebe on Wednesday December 25, 2024 @02:25PM (#42510608)
I've been using Thunderbird with GMail for 10+ years. The main issue was having to switch to oauth login a while back.
Re:Ten Years of JMAP by amluto on Tuesday December 24, 2024 @11:41AM (#42502997)
I have no idea whether JMAP is any good, but I find all the hype utterly baffling. JMAP is supposedly awesome. There is real thoughtful engineering going into offline support: https://www.fastmail.com/blog/offline-mail-storage/

And yet: at least as of a couple months ago, there isn’t actually any offline support! The Fastmail first party app does not work at all offline. Complete failure. Does not even try. Oh, and it has dramatically worse threading support than even Gmail or Outlook.

I don’t get it. POP3 is awful and kind of works offline. IMAP is old, clunky, and works fairly well offline. JMAP is supposedly the new hotness, but the paid first party experience will not even try to load offline. This is table stakes! Eudora could do this. Every version of Outlook ever could do this. Thunderbird can, and always could, do this. The usual command-line clients work offline if configured appropriately. Heck, Google went above and beyond and made Gmail’s webmail work quite well offline if you care to set it up, and I think they did a bunch of early work on service workers to make this possible. Fastmail, please stop pitching your fancy protocols until you can get your clients up to the state of the art as of twenty to thirty years ago.

Re:Ten Years of JMAP by radicality on Tuesday December 24, 2024 @02:35PM (#42504312)
Similarly, I wish I could find some iOS mail app that allows me to have the have the full mailbox locally. I use Gmail with the Mail app on iOS, and tried multiple mail apps to try to find one that will download the whole mailbox with all attachments and let me search it locally quickly, but seems that doesn’t exist. On Mac I’m using Mailmate and liking it.
Re:Ten Years of JMAP by Mailtemi on Thursday December 26, 2024 @12:15AM (#42513285)
I'm actually working on this (https://mailtemi.com/blog/storage-redesign/). After pushing Mailtemi on Android, I will continue working on full offline support.
Re:Ten Years of JMAP by radicality on Friday December 27, 2024 @07:14AM (#42521566)
That’s awesome, will check it out!
Re:Ten Years of JMAP by Mailtemi on Friday December 27, 2024 @01:24PM (#42524604)
This is just a step toward offline support. It's not available yet, because Android comes first.

The app is ready for offline use, but I don't want to implement it without 'full text search.' I plan to use SQLite FTS5. My initial tests with search show that offline search is storage-expensive, which could be problematic for mobile devices. So, the edge cases have started to appear. But it's the most requested feature, and the most interesting/challenging to me :)

Re:Ten Years of JMAP by radicality on Friday December 27, 2024 @02:40PM (#42525348)
Yeah for high-end phones with lots of storage, fully offline full text search would be really nice. Often I know that some email with a substring exists and I search for it in Mail app, and it goes off to do a server search… And when you need it most, you probably have 1 bar of range, or have no cell service and no available WiFi. Example: pulling up reservation details while traveling in a foreign country.

Or even worse, sometimes I look at some email in Mail and it’s there, and then hours later, while on bad network conditions, I look at it again and see “This email has not been fetched from the server. Download?”, which then just hangs.

I have a 1TB iPhone and a 2TB iPad. My main Gmail account is not large by any means - ~50k emails and 5GB space. Even though I have all this space on my iPhone/ipad available and willing to commit a bunch of it to email with all attachments and full text index, there seems to be no way to do it.

The Gmail app has a slider for “days of mail to sync”, but the max is 90 days, and not “All time” like one would expect. Anyways, good luck with Mailtemi!

Re:Ten Years of JMAP by antifarben on Thursday December 26, 2024 @02:44AM (#42513719)
StalwartMail supports JMAP out of the box