News for hackers. Stuff that matters.
Re:Show HN: I'm working on an open-source Gmail replacement by vidarh on Wednesday November 06, 2013 @06:29AM (#6682134)
> I wonder when filesystems became good enough for maildir to work with your volume of emails.

'99-2001 I designed and deployed a webmail service where we handled ~2 million user accounts using Maildir's on a single SAN device formatted with ReiserFS (the fridge sized device in question was not much faster or higher capacity than my current laptop.... Actually the whole infrastructure was probably not much higher capacity than my laptop). Many of us working on it had individual folders with tens of thousands of messages.

"Pure" Maildir without the client doing additional indexing will quickly get very IO intensive, but storing hundreds of millions of small files in a single filesystem was a solved problem by then and with some simple indexing or caching its trivial to handle that volume.

We handled the indexing needed to reduce the IO load imposed by the clients by writing a custom POP server and delivery agent that maintained some more metadata - partially in the filename of the messages (read/unread status etc.) - saving us a stat() call per message on scanning the folders. Partially by maintaining cache files with even more details, saving us re-scanning the message headers (accessed by the web frontend via a couple of custom/non-standard POP commands - much easier than contemplating dealing with the horrors of IMAP; for actual non-web client access we put proxies in front of our "special" POP server).

Re:Show HN: I'm working on an open-source Gmail replacement by eru on Wednesday November 06, 2013 @07:49AM (#6682401)
Thanks for the report. Do you think maildir is the way to go for storing email (plus additional indices), or do you think a more custom format (eg in some relation database or something completely different) is better?
Re:Show HN: I'm working on an open-source Gmail replacement by vidarh on Wednesday November 06, 2013 @08:44AM (#6682595)
I prefer Maildir but mostly because it is trivial to manipulate because it is so simple. Our design back then was based on the assumption that we could blow away all the cache files at any time, and it could all be trivially rebuilt from just the raw messages.

I still have a soft spot for flat files when possible for that reason.

Also for our usage we saw a relational DB as a scalability hassle, as we needed to be prepared to scale this to many backends, and dealing with an RDBS when each set of objects was specific to a user was not very appealing. Today the challenge in that is much smaller due to faster hardware, and there are far more options in terms of database solutions.

But at the same time there's just not that much to gain.

Re:Show HN: I'm working on an open-source Gmail replacement by eru on Thursday November 07, 2013 @03:58AM (#6688381)
Yes, I like maildir, too, because I can use all the well known unix tools (like grep) to manipulate and filter my emails.