News for hackers. Stuff that matters.
Re:Show HN: I'm working on an open-source Gmail replacement by tluyben2 on Wednesday November 06, 2013 @03:00AM (#6681473)
I welcome this as I welcome Mailpile and other efforts like it; I look for another mail solution every few years. Not with the idea of replacing gmail (might/might not do), but to see if anyone did anything useful yet. So far, no. All solutions, even considering the changes to gmail everyone seems to hate (I don't), are nothing compared.

Every time some new frontend/backend comes up, I say the same (and have done on HN before); i'm not a typical user; I had email since '95, I have been a heavy user since then and I took my mail with me since. In 2005 I was looking, like I had been then as well, for a better mail solution and stumbled upon gmail. I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail.

I now have over a million mails in my mailbox (i'm not sure what is the total); I have over 50 mail addresses coming to my inbox, I receive 1000s (sometimes 10.000s) of spam messages per day which Google filters well. I guess this is due to the fact I have had/have businesses on those 50 mails since 1995. All clients I tried so far just simply hang when I try them, including outlook (exchange or imap), thunderbird and some free and commercial web versions (yahoo and outlook web/live simply don't work; I cannot even read my mail through the amount of spam and the clients are horrible for productivity imho. Slow as well).

I also need a solid spam solution; spamassassin simply doesn't cut it; not only does it run high processor on my server, it doesn't actually filter stuff like google does. Google almost never goes wrong for me; actually; I have had very few mistakes / missing mails. While with spamassassin, I'll be carefully inspecting 5k mails / day while still getting spam in my inbox.

I think my mailbox is a bit weird now, but it'll be quite normal as it'll be normal for people to have a mailbox since birth and taking it with them till they die. If you run a few businesses along the way, getting to 1 million messages is not hard; spam will find you as well. Media messages are getting common; I make a point of using dropbox/sftp for attachments, but not everyone does that, so I do get videos, huge blah megapixel cameras of birthdays of family etc. This is normal and will only grow; the current mail solutions don't handle it well. If you want to deliver a competitor to gmail, you need to make this work imho and it needs to be a test case.

Re:Show HN: I'm working on an open-source Gmail replacement by justinmk on Wednesday November 06, 2013 @03:12AM (#6681519)
Yeah, the gmail UI (which is forgettable) is about 5% of the product; the other 95% is:

* spam detection

* search

* mail header hacks

I don't think that can be approximated in "a few months".

> 50 mail addresses coming to my inbox

On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases).

Re:Show HN: I'm working on an open-source Gmail replacement by tluyben2 on Wednesday November 06, 2013 @03:20AM (#6681546)
Like I put in the tail of my post; I don't think i'm an outlier anymore soon. The 50 email addresses; sure, but the amount of mail, maybe for now, but not for long. And certainly not (already!) in size; there are plenty of people who get/send all their video/images/large docs via email.

Edit: also, the Google team was really good to me and my (free! well I pay more for storage obviously, but it's not a google apps one) mailbox; on several occasions they contacted me when my mailbox was not functioning and helped fix the issues with it. Compared to a few years ago, the gmail backend, for me has gotten a lot better; search used to just stall; now it's always fast and accurate. And when I deleted a significant amount of mails, my mailbox would be unreachable for at least 1 day (giving a Oops error code 14 or something like that); no it just works.

Edit2: I rather would like the spam AI from Google as a webservice for all clients to use, or an open source alternative. Spamassassin is a nice initiative, but it never worked for anyone I know. A lot of my colleagues just ended up routing their company mail through gmail to get a better spam filter.

Re:Show HN: I'm working on an open-source Gmail replacement by 300bps on Wednesday November 06, 2013 @07:17AM (#6682292)
You are an outlier and will always be an outlier. I've had email since I got my first modem in 1985. I've had hundreds of email addresses between BBSes, online services and the Internet. I've happily deleted all the mail and given up the addresses for all but a handful.

What might change at some point is A&E launching a Digital Hoarder reality show wherein people with such a storage proclivity can get the help they need.

Re:Show HN: I'm working on an open-source Gmail replacement by tluyben2 on Wednesday November 06, 2013 @07:39AM (#6682355)
That would be one massively boring show :)

Maybe you are right. I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway (considering the allotted yearly storage growth and the plummeting prices of storage in general) you won't bother deleting anything.

You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes. I ran my BBSes, for most of the 80s, of 720 kb disks, so I do know what storage limitation was. It's just not there anymore. Gmail now easily handles all of this and so should some other solutions imho, especially the ones wanting to compete with gmail. Google said; 'never delete another mail' and i'm sure people got used to that by now for all cloud services.

The mail address and email hoarding came when I started making serious companies. And it's not like they bother me or whatever. Maybe I will throw everything away some time, but for now; I dodged an almost million euro gov fine a few years ago by having a complete mail conversations + attachments with my employees between 2002 and 2004. I would be in a whole load of crap if I wouldn't have those anymore.

Re:Show HN: I'm working on an open-source Gmail replacement by pieterhg on Wednesday November 06, 2013 @07:55AM (#6682423)
I think you're absolutely right. We've actually been thought by Gmail not to remove messages but instead archive them and that's a good philosophy.

I've been on Gmail since 2005 and have about 100,000 messages at 20 GB. There's messages coming in from 25 different addresses at 300/day with 100 being spam.

I don't feel like I'm an email power user though, I imagine people in sales do about 100x more than this.

It's definitely a requirement to be able to do this and I feel even if we would be outliers, in a couple of years we won't be anymore.

Re:Show HN: I'm working on an open-source Gmail replacement by tluyben2 on Wednesday November 06, 2013 @09:24AM (#6682778)
And at least Google seems to think so; they worked with me to get my mail fast and working well. They asked me to turn stuff on and off, reroute addresses for a while and such until my mail (as it is now) was fast and solid. I think they do expect a lot of people to be like me in the future.
Re:Show HN: I'm working on an open-source Gmail replacement by 300bps on Wednesday November 06, 2013 @07:54AM (#6682418)
You said the following two things which are mutually exclusive:

I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail.

I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway

The default status is that stuff goes away. I believe you're arguing that "this time it's different" but I see no evidence for that. In fact, there is a post almost every day on HN about Gmail users looking for an alternative. I myself can't wait to get off Gmail. When I do that I'll happily explicitly delete my Gmail account and all the email in it. Because I am lazy and it's easier to do that than it is to write a script to transfer it to what I use next.

You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes.

I got my first modem in 1985, got a phone line as my Christmas present from my parents in 1986 and started my own BBS. Started a consulting company in 1993. Started a small ISP in 1996, sold it in 1998 and started working for corporate America.

In the U.S., I find from a legal perspective that the less documentation you have the better. That's why every large U.S. company has an email retention policy that is very short. Anyone looking to sue you needs proof and often that proof comes from your own records as part of discovery since they don't have any proof. If as part of a standard process you delete everything you're not legally required to retain then you can't be forced into helping their case.

Re:Show HN: I'm working on an open-source Gmail replacement by tluyben2 on Wednesday November 06, 2013 @09:21AM (#6682763)
> I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail. > I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway

My, old person who grew up in the 80s, case versus people I know and talk to who are <= 20 y/o now.

> In the U.S., I find from a legal perspective that the less documentation you have the better. etc

This particular thing was a large project, paid for by a gov grant (not sure if that's the correct word, but I think it means the same) early and ran from 2001-2009; the grant was for millions. A few years ago, suddenly they came to check on the project; see if all paperwork is in order, hours written etc. Now during one of the office moves, someone misplaced the actual paperwork we prepared for this including CDs with source code, revision history etc (Now only if there was some kind of storage on internet to store this on...). We would have to pay that money back + fines if we cannot actually prove we did the work and who did it. In your case, we would've had to pay the money/fines, because outside my mail, there was no way to prove anything. And these gov checkups come years and years after the fact.

Granted; this was a stupid mistake of someone who thought the boxes were full of outdated crap, maybe even an external mover, however, I don't see the downside with keeping everything in this case. Especially if you have a company. In the US you have all kinds of legal issues; I understand that, we don't have that really, so worrying that someone will check your mail, even in courtcases, is not very likely.

Re:Show HN: I'm working on an open-source Gmail replacement by hrjet on Wednesday November 06, 2013 @06:18AM (#6682099)
My hunch is that Google's spam detection is not necessarily better than open-source solutions. It's just that they may have a larger corpus of data to train the system on.

If I host my own email server, my spam filter will only have my inputs to determine what is spam.

This leads me to an idea. There probably can be a service to build a dynamic corpus of spam samples. If I mark a message as spam, my email server can send a sample to the service, and periodically, my email server will download "top 100" samples to feed its filter.

Re:Show HN: I'm working on an open-source Gmail replacement by kilburn on Wednesday November 06, 2013 @06:45AM (#6682197)
This isn't necessarily a good idea. The type of SPAM you receive is different than the type of SPAM I receive. Moreover, it is very possible that I do (not) consider some messages as SPAM whereas you do (not). It is true that it takes some time to train the filter, but after that initial period you will get better results if you only train it with your own SPAM.

I've been hosting my own email on a small personal server for some ~7 years now. All my email addresses redirect here (I have 9 frequently used addresses, and a bunch of website-specific ones). I'm using spamassassin (plus amavisd-new) as a proxy content filter. These are my filter's numbers for the last three days:

      2 BAD-HEADER-2
      1 BAD-HEADER-8
     47 BANNED
    297 CLEAN
      6 INFECTED
    426 SPAM
     14 SPAMMY
Spammy mails are delivered to a special "Spam" folder (just like in GMail). All those were indeed SPAM messages. I've received no spam messages outside of those. The high number of CLEAN mails is because I'm subscribed to a few mailing lists and I receive notifications/tracking info/alerts from a number of services.

TL;DR: SpamAssassin is hard to setup and train correctly (just like the rest of a server's mail-stack). However, once you get over this part, the results can be as good as gmail's.

Re:Show HN: I'm working on an open-source Gmail replacement by justinmk on Wednesday November 06, 2013 @09:55AM (#6682946)
> I've received no spam messages outside of those.

How protective are you of your email address? My email is posted all over the internet, but I haven't seen spam in gmail for at least 6 months--probably longer.

Re:Show HN: I'm working on an open-source Gmail replacement by enjo on Wednesday November 06, 2013 @11:58AM (#6683693)
I have a huge spam problem with gmail. Someone sold my email + first name. Googles spam filters are simply powerless if you include a first name in the email. I've reported these as spam (several a day) for months at this point.

Hell, it gets marked as important.

Re:Show HN: I'm working on an open-source Gmail replacement by magicalist on Wednesday November 06, 2013 @04:16PM (#6685558)
> Googles spam filters are simply powerless if you include a first name in the email

I think you're going to have to look elsewhere for a cause. Someone (probably several someones) sold out my email and first and full name. Gmail has no problem correctly categorizing all sorts of spam addressed to me by name.

Re:Show HN: I'm working on an open-source Gmail replacement by kilburn on Thursday November 07, 2013 @03:27AM (#6688290)
I may have an explanation for that. Gmail is very good at detecting waves of spam mails while the wave is being sent out. Basically, a (bunch of) hacked server(s) start sending SPAM mails. After some messages google realizes that these servers have been hijacked and starts rejecting mails from them.

If your address happens to be among the first of the spammer's list, you will be hit by these emails. Otherwise when they send to your address google has already stopped accepting mails from that (those) ip(s) and you don't receive anything.

It looks like enjo's address comes way before than yours...

Re:Show HN: I'm working on an open-source Gmail replacement by baudehlo on Wednesday November 06, 2013 @05:31PM (#6685979)
Whatever the cause is, it's still Google's fault/problem.

I too have a big gmail spam problem (related to my email address being very short and guessable), and mark as spam on average about 10-20 emails a day.

Re:Show HN: I'm working on an open-source Gmail replacement by ChristianBundy on Wednesday November 06, 2013 @08:33PM (#6686890)
> Whatever the cause is, it's still Google's fault/problem.

Why is it that no matter what the cause is, you feel that Google is to blame?

Re:Show HN: I'm working on an open-source Gmail replacement by dbpatterson on Wednesday November 06, 2013 @09:39PM (#6687127)
Err... because Google's spam filter is letting the message in? Seriously, it is a spam message, delivered to Google, and Google put it in the inbox instead of the spam filter. I don't know how much more clear cut it could be.
Re:Show HN: I'm working on an open-source Gmail replacement by kilburn on Wednesday November 06, 2013 @11:51AM (#6683651)
I have a special "don't care" mail account whose mails are delivered to its own folder (included in the shown stats). I do get a spam message in that folder very occasionally (as in once per month), but it's really far off from becoming annoying.
Re:Show HN: I'm working on an open-source Gmail replacement by oijaf888 on Wednesday November 06, 2013 @01:11PM (#6684216)
Isn't that pretty much what Vipul's Razor does? http://en.wikipedia.org/wiki/Vipul%27s_Razor
Re:Show HN: I'm working on an open-source Gmail replacement by hrjet on Wednesday November 06, 2013 @10:12PM (#6687207)
Wasn't aware of that; thanks!
Re:Show HN: I'm working on an open-source Gmail replacement by snogglethorpe on Wednesday November 06, 2013 @06:27AM (#6682125)
... and on top of all that is the simple fact that gmail is backed by a very robust infrastructure and supported by company that's very unlikely to go out of business anytime soon. [You can make jokes about Google reader etc, but gmail is far too important to Google's public image to make that a reasonable concern.]

Together with the very decent UI, great spam handling, and free space, this makes gmail a pretty good choice for a lot of people.

Even with a good free version of the software, duplicating the entire package is a huge task ...

Re:Show HN: I'm working on an open-source Gmail replacement by e12e on Wednesday November 06, 2013 @03:23AM (#6681562)
> On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases).

I think (hope!) there's room for niche products. I would never try to make an email client "for everyone". All the ones I've tried are crap (because they're "for everyone").

If you'd said: define your market and optimize for that - then I'd agree. But to not aim for a niche if you're doing a small project sounds just crazy (and wrong).

Re:Show HN: I'm working on an open-source Gmail replacement by justinmk on Wednesday November 06, 2013 @10:02AM (#6682992)
> define your market and optimize for that

Right. But if you're writing a gmail clone, just don't get sidetracked by the token comment about email archives from 1985.

Re:Show HN: I'm working on an open-source Gmail replacement by vincentkriek on Wednesday November 06, 2013 @03:20AM (#6681542)
What do you mean with mail header hacks?
Re:Show HN: I'm working on an open-source Gmail replacement by justinmk on Wednesday November 06, 2013 @10:07AM (#6683021)
Mail headers and mail relays are notoriously inconsistent.

http://www.jwz.org/doc/threading.html

Re:Show HN: I'm working on an open-source Gmail replacement by eitland on Wednesday November 06, 2013 @11:00AM (#6683320)
Thanks for posting the link!
Re:Show HN: I'm working on an open-source Gmail replacement by vpeters25 on Wednesday November 06, 2013 @01:27PM (#6684375)
Search is my biggest issue too. I switched out of gmail to my self-hosted server using roundcube and searching through IMAP is excruciatingly slow compared to gmail, even after I moved the mailboxes to an SSD.

I am not sure whether this is a limitation on the IMAP protocol, but I suspect a proper replacement to gmail might involve coding a new mailbox backend and protocol.

Re:Show HN: I'm working on an open-source Gmail replacement by brdrak on Thursday November 07, 2013 @06:04AM (#6688757)
I may be wrong, but I believe IMAP, the protocol, just defines how IMAP clients/servers should issue and respond to search commands, but doesn't deal with mail indexing and things like that.

Cyrus IMAP, which I use, provides Squatter, which substantially speeds up search, but the IMAP client is still the interface. Performance I get is around a sec for a simple query, a few sec for a more complex one against a folder with 20K messages.

Last I checked, Roundcube had very limited search capabilities. Thunderbird is much better, but I'd prefer a terminal based search. Something like SQL. Does anyone know of a tool like that?

Re:Show HN: I'm working on an open-source Gmail replacement by morkbot on Wednesday November 06, 2013 @04:48AM (#6681834)
Add to that the speed. Roundcube looks pretty good, even Horde's could make a cut GUI-wise (judging by screenshots from the latest versions), but they both lag in the speed department, comparing to GMail or Fastmail.
Re:Show HN: I'm working on an open-source Gmail replacement by mbesto on Wednesday November 06, 2013 @10:02AM (#6682989)
I would happily pay someone non-google to get those 95% things right and then add in mobile sync, gmail-like interface, and you've got a customer.
Re:Show HN: I'm working on an open-source Gmail replacement by sp332 on Wednesday November 06, 2013 @03:32PM (#6685308)
What parts of gmail's webmail interface do you find indispensable?
Re:Show HN: I'm working on an open-source Gmail replacement by massysett on Thursday November 07, 2013 @09:32AM (#6689474)
For me, nothing; the huge thing about Gmail is it works flawlessly from my Android phone. No IMAP solution came close. These days I read most of my mail on the phone.

Once I was in a long check-in line at the airport. Silly me, I figured I wouldn't check in at home because I had to check bags. At the airport I got on my phone, pulled up the confirmation email, checked in, and went to the bag drop line. With the clumsiness of IMAP and dealing with it on phones I would never have managed to do this if I was hosting my own mail, and even if I had managed to do it, the cost would have been the hours and the hassle of maintaining mail.

It was at that moment that I realized I would never host my own mail again. Just not worth it.

Re:Show HN: I'm working on an open-source Gmail replacement by Wingman4l7 on Wednesday November 06, 2013 @04:26AM (#6681753)
The insufficiency of the alternative spam solution is what I was afraid of. =/ It's easy to forget how awesome Gmail's spam filter is when you get spam in your inbox so rarely that you almost forget it exists at all.

I keep saying that Google could make a mint if they licensed an API for it. They probably just don't want to bother and would prefer that they keep its talents exclusive to Gmail. Also, I suppose that the people who'd want to utilize it would be leery of running their mail through Google's servers.

Re:Show HN: I'm working on an open-source Gmail replacement by csmuk on Wednesday November 06, 2013 @03:55AM (#6681660)
I've been using mutt for 16 years and self hosting email. It's currently hosted on bigv.io. Spam is handled by spamd (ships with OpenBSD) and SMTP by OpenSMTPd. Local delivery is via procmail.

I have 40,000 messages archived with attachments etc.

Granted it's not pretty but it's fast, efficient, can read html mail fine and is pretty immune to spam.

The best thing is I haven't changed it for years.

Re:Show HN: I'm working on an open-source Gmail replacement by ape4 on Wednesday November 06, 2013 @11:13AM (#6683398)
I admire your purity. But what about viewing images in a mail?
Re:Show HN: I'm working on an open-source Gmail replacement by csmuk on Wednesday November 06, 2013 @11:38AM (#6683568)
Easy. Add to .mailcap:

   image/*; yourviewer %s
Then hit "v" on the image attachment and it pops up in a window. I use xv (still) to view it which runs under X. Hit "d" to download it.

I handle HTML mail, PDFs, word documents, everything with complete ease.

Re:Show HN: I'm working on an open-source Gmail replacement by brdrak on Thursday November 07, 2013 @05:43AM (#6688711)
Weird, I have exactly the opposite experience with spam -- my self hosted setup does better than gmail. Wrote this a while back, but still holds true: https://news.ycombinator.com/item?id=5441542

I don't have nearly the volume of email you do though. My inbox has 20K, probably another 70K in other folders. Thunderbird is fine for me, except I'd like a more powerful and faster search. Gmail search is not good enough, though much better than other webmail clients I've tried.

Re:Show HN: I'm working on an open-source Gmail replacement by sp332 on Wednesday November 06, 2013 @03:37PM (#6685341)
I have had very few mistakes / missing mails.

It's still pretty good, but I have been getting a lot more false positives in gmail lately than I used to. Even a security code from hotmail got redirected to spam, which I thought would be common enough (and important enough) that Google would have learned about it already.

Re:Show HN: I'm working on an open-source Gmail replacement by eru on Wednesday November 06, 2013 @03:13AM (#6681522)
Just out of interest: do you know if mutt (plus maildir and, say, getmail) cope with the volume?
Re:Show HN: I'm working on an open-source Gmail replacement by tluyben2 on Wednesday November 06, 2013 @03:24AM (#6681567)
I used mutt on the server as email client before switching to gmail. Simply because no desktop client would cope. So at least till 2005 it was fine with mutt; I had to ssh (that was fine), spam issues (nothing to do with mutt, but still) and worse; I was running a business; attachments were a big issue. I had to save them and scp them over. Later I just created a mailbox binaries@... to which I forwarded the mails with attachments I wanted to see, which I then retrieved with a pop3 desktop mail client. It's just not very practical when running (a) business(es).

I would imagine there are better solutions for the attachments now; in 2005 you couldn't really find that much to solve it as I remember.

Re:Show HN: I'm working on an open-source Gmail replacement by icebraining on Wednesday November 06, 2013 @04:33AM (#6681777)
I had to save them and scp them over.

I just mounted my ~/mail/attachments locally using SSHFS. Saving them with Mutt made them automatically "appear" on my machine. You could also install a few viewers on the server and then use X11 forwarding, which works fine if you have a decent connection.

Re:Show HN: I'm working on an open-source Gmail replacement by eru on Wednesday November 06, 2013 @03:31AM (#6681581)
Thanks. I guess mutt plus bayesian spam filter might work reasonably well.

I wonder when filesystems became good enough for maildir to work with your volume of emails.

Re:Show HN: I'm working on an open-source Gmail replacement by vidarh on Wednesday November 06, 2013 @06:29AM (#6682134)
> I wonder when filesystems became good enough for maildir to work with your volume of emails.

'99-2001 I designed and deployed a webmail service where we handled ~2 million user accounts using Maildir's on a single SAN device formatted with ReiserFS (the fridge sized device in question was not much faster or higher capacity than my current laptop.... Actually the whole infrastructure was probably not much higher capacity than my laptop). Many of us working on it had individual folders with tens of thousands of messages.

"Pure" Maildir without the client doing additional indexing will quickly get very IO intensive, but storing hundreds of millions of small files in a single filesystem was a solved problem by then and with some simple indexing or caching its trivial to handle that volume.

We handled the indexing needed to reduce the IO load imposed by the clients by writing a custom POP server and delivery agent that maintained some more metadata - partially in the filename of the messages (read/unread status etc.) - saving us a stat() call per message on scanning the folders. Partially by maintaining cache files with even more details, saving us re-scanning the message headers (accessed by the web frontend via a couple of custom/non-standard POP commands - much easier than contemplating dealing with the horrors of IMAP; for actual non-web client access we put proxies in front of our "special" POP server).

Re:Show HN: I'm working on an open-source Gmail replacement by eru on Wednesday November 06, 2013 @07:49AM (#6682401)
Thanks for the report. Do you think maildir is the way to go for storing email (plus additional indices), or do you think a more custom format (eg in some relation database or something completely different) is better?
Re:Show HN: I'm working on an open-source Gmail replacement by vidarh on Wednesday November 06, 2013 @08:44AM (#6682595)
I prefer Maildir but mostly because it is trivial to manipulate because it is so simple. Our design back then was based on the assumption that we could blow away all the cache files at any time, and it could all be trivially rebuilt from just the raw messages.

I still have a soft spot for flat files when possible for that reason.

Also for our usage we saw a relational DB as a scalability hassle, as we needed to be prepared to scale this to many backends, and dealing with an RDBS when each set of objects was specific to a user was not very appealing. Today the challenge in that is much smaller due to faster hardware, and there are far more options in terms of database solutions.

But at the same time there's just not that much to gain.

Re:Show HN: I'm working on an open-source Gmail replacement by eru on Thursday November 07, 2013 @03:58AM (#6688381)
Yes, I like maildir, too, because I can use all the well known unix tools (like grep) to manipulate and filter my emails.