News for hackers. Stuff that matters.
Re:Decommission of Cyrus Email by colechristensen on Saturday May 06, 2017 @07:38PM (#14282925)
It's really disappointing that nothing better than gmail has come along.

There _are_ open source alternatives, but they aren't as good, that's all there is to it. Setting up a mail server is hard, not because it needs to be but because the software requires eldritch incantations to make it work properly (there are too many things to keep track of in order to A) get email delivered at all and B) not become a spam relay)

It's possible to be something in between the "Just Works" walled garden that gives you no options or control and the archaic wizardry which requires a 2 foot grey beard to understand.

Re:Decommission of Cyrus Email by paulddraper on Saturday May 06, 2017 @07:41PM (#14282936)
For me, the killer feature of Gmail that's hard to compete with is spam filtering.

I remember when people thought spam was single handedly going to make email unusable.

Re:Decommission of Cyrus Email by Tharkun on Saturday May 06, 2017 @08:01PM (#14283009)
On the other side of that, sending mail to gmail can be rather challenging. They have a tendency of 541-ing mails for no apparent reason, under the pretense of "poor mail server reputation".

Basically you don't control who gets to send you mails. You're entirely at Google's uncompromising and unreachable mercy.

Re:Decommission of Cyrus Email by colechristensen on Saturday May 06, 2017 @08:21PM (#14283102)
If you run DKIM, SPF, and DMARC, you shouldn't have any delivery problems.

DKIM is signing your email with the public key in DNS.

SPF is whitelisting which mail servers send mail from your domain

DMARC is how receiving ends should report how your mail is delivered (or not).

Unless you're actively spamming (ahem, "marketing") to large volumes of users, you should have no trouble at all getting mail delivered to users.

If you are doing mass email marketing, stick with your core competence and let someone else handle the mail servers.

Re:Decommission of Cyrus Email by interfixus on Saturday May 06, 2017 @11:58PM (#14283771)
You shouldn't have any problems. But you do.

After fifteen years running my own mail systems, I have finally given up. Ever more outgoing mail was getting lost, despite everything being up to date and done by the book. The arch villains, over and over, were Gmail and the MS thing - Hotmail, outlook.com, Live, whatever they call it.

Re:Decommission of Cyrus Email by techsupporter on Sunday May 07, 2017 @04:09AM (#14284407)
> After fifteen years running my own mail systems, I have finally given up.

Same here. I've been hosting my own e-mail for two decades (as of this year) and I gave up earlier this year. My spouse's e-mail to friends who use GMail--something like 80% of them--would, every couple of months, disappear into the recipient's spam folder or sometimes just disappear entirely. My logs would show a successful delivery but the recipient couldn't find the message anywhere. Occasionally messages would start going into junk in the middle of a conversation.

Nothing I changed would permanently fix it. I'd send from a different IP in my block (something.something.194 instead of something.something.193) and mail would flow again...for a couple of months and then stop. My domain registration predates Google's existence by 18 months. I've been sending from the same IP subnet for six years. My colo provider is actually so small that I'm the only customer left on my "neighborhood" /24 (they've recently asked me if I want the entire block routed to me for the lulz). I have SPF, DKIM, and DMARC all set up and they've not changed for at least two years. The amount of mail outbound from my server is (was?) so little that I could tail the outbound mail log and keep up with each message being sent.

None of that mattered. I even signed up for Google's Postmaster Tools but, and this really cooked my goose, I send so little e-mail that I don't qualify to show up on their reports. I tried contacting Google but hahahahahahahahahahahaha--wheeze--hahahahahahahaha.

No other recipient had a problem with me. Messages were successfully delivered to Microsoft and Yahoo just fine. Only Google had a problem with my e-mail.

I finally caved. Google has SO MUCH of the e-mail market that unreachability to them is a catastrophe...and they know it. I don't have the leverage to make them play nice.

Thankfully, Fastmail does so I've changed to them. For the first time since February 1997 my MX doesn't point to my own server and that makes me a little sad.

Re:Decommission of Cyrus Email by interfixus on Monday May 08, 2017 @12:35AM (#14289283)
My experience exactly, though I didn't bother to spell it out in detail. Also the final move to Fastmail, who, it must be said, serves me nicely and well.
Re:Decommission of Cyrus Email by techsupporter on Monday May 08, 2017 @09:46PM (#14297149)
Yep, if it weren't for Fastmail I'd still be hosting my own e-mail and just quietly cursing the existence of Gmail. Now I get to have high significant other acceptance factor and curse the existence of Gmail.
Re:Decommission of Cyrus Email by JoshTriplett on Saturday May 06, 2017 @09:32PM (#14283355)
If you're not careful with how you do those things, though, then you'll be unable to send mail through quite a bit of standard mailing list software. It's possible to configure some (not all) of those in a way that allows mailing lists to work, but some of them are just completely incompatible.
Re:Decommission of Cyrus Email by colechristensen on Saturday May 06, 2017 @10:15PM (#14283477)
True. When it comes down to it, I'm on the side that says mailman is doing wrong when it modifies messages breaking their signatures.

Here's some background https://wiki.list.org/DEV/DKIM

Re:Decommission of Cyrus Email by bradknowles on Sunday May 07, 2017 @12:18AM (#14283824)
The people who designed DKIM didn't care about how mailing lists worked, they just did it anyway. But Mailman has adapted over the years, and Mailman3 is greatly improved. IMO, it's the best mailing list package out there, especially with regards to DKIM and DMARC compliance, but only if it is configured correctly.

However, even that can only carry you so far. Gmail is screwed up in how they handle reputation, especially with regard to IPv6. The exact same message delivered to Gmail via IPv6 will get refused versus sending it via IPv4 will work fine.

This problem has existed for years. There are even standard hacks for postfix to cause all mail addressed to Gmail to be sent via IPv4 only, just to avoid this problem.

And any domain hosted with GAFYD is likewise screwed.

You can't know what your false positive stats are, because you don't have access to all the remote SMTP spools or logs on the 'net. All you can know for sure is the false positives that you find out about. But you literally have no way to tell how big the iceberg is under the water.

Re:Decommission of Cyrus Email by lloeki on Sunday May 07, 2017 @03:07AM (#14284263)
> But you literally have no way to tell how big the iceberg is under the water.

Aren't DMARC reports supposed to tell you that, especially ruf?

Re:Decommission of Cyrus Email by tankenmate on Sunday May 07, 2017 @05:34PM (#14287816)
I've found DMARC reports to be incomplete or flat out lies from time to time.
Re:Decommission of Cyrus Email by JoshTriplett on Saturday May 06, 2017 @11:25PM (#14283680)
I'm not a big fan of mailing list software that modifies the non-header parts of messages. But, for instance, some of the signature schemes utterly break the ability of a mailing list to send mails on to the rest of the list members while preserving all the headers.
Re:Decommission of Cyrus Email by andybak on Saturday May 06, 2017 @08:14PM (#14283066)
Is this bulk mail? (I'm not implying spam - transactional or opt-in)

I only ask because I don't recall any problems over the years with one to one emails either to my gmail or from me to other people's gmail accounts.

Re:Decommission of Cyrus Email by peterwwillis on Saturday May 06, 2017 @08:59PM (#14283236)
AOL and Yahoo and a bunch of other providers used to be tough as well. I don't think there is much benefit to being _less_ scrutinizing when your user base is mostly consumers.
Re:Decommission of Cyrus Email by skarap on Sunday May 07, 2017 @07:28AM (#14284932)
> They have a tendency of 541-ing mails for no apparent reason

There is a reason. It makes people switch from own email servers to gmail.

Re:Decommission of Cyrus Email by vidarh on Saturday May 06, 2017 @10:08PM (#14283460)
Spam filtering is the "killing" features for my Gmail use. It traps way too much legitimate e-mail for me, to the point where I know I need to move off Gmail, it's just a matter of finding the time - I can't trust it any more.
Re:Decommission of Cyrus Email by acdha on Saturday May 06, 2017 @10:11PM (#14283470)
That's really curious - my false-positive rate is really low (I do review my spam folder) and often arguably correct (e.g. legit senders who've hosed their SPF settings).

Do you have an idea what's triggering the false positives – keywords, IP reputation, etc?

Re:Decommission of Cyrus Email by vidarh on Sunday May 07, 2017 @12:17AM (#14283818)
I see a wide variety. E.g. anything that can generate bursts of mails, such as alerting tends to regularly end up in the spam folder and that's particularly problematic. And it keeps happening even though the sending addresses is in my contact list and I've repeatedly pressed "not spam". Newsletters I've signed up to and where they keep ending up in the spam folder despite me repeatly pressing "not spam". Personal, individual e-mail messages sent from Gmail by freinds. I've even had messages generated by Google get caught by the spam filters, including idiotically enough moderation reports from Google groups.

The overall rate relative to my total e-mail volume is quite low, but it's enough that I have to check far more often than I'd like.

It wouldn't be a problem if I had a reliable way of whitelisting addresses, but none of the methods seems to be consistently reliable.

Re:Decommission of Cyrus Email by lloeki on Sunday May 07, 2017 @03:02AM (#14284252)
I moved to FastMail lately and was pleased to discover that you could have some control over the spam filter[0], as well as useful advice:

    Note: We recommend that you do not mark your Spam/Junk Mail folder to
    automatically learn "As spam". This can create a false positive feedback loop.
    Imagine an email is incorrectly classified as spam, put in your Spam/Junk Mail
    folder, and then learned as spam. That means future emails that aren't spam are
    now more likely to be incorrectly marked as spam, sent to your Spam/Junk Mail
    folder, and learned as spam. Only mark folders to learn "As spam" if they're
    folders you manually move email to.
Now on top of that say you're using a MUA and its own spam filter runs atop of that, and decides to move a couple of legit stuff to Gmail's spam folder, triggering the server-side spam learning process. Surefire way to screw the whole thing up (happened).

[0]: https://www.fastmail.com/help/receive/stopspam.html?u=b83140...

Re:Decommission of Cyrus Email by acdha on Sunday May 07, 2017 @10:14AM (#14285545)
Interesting - do you use the priority inbox feature? I have an extremely low false-positive rate for things like newsletters but do have some other things incorrectly marked as Promotions.

I've found the never mark as spam filter option to be reliable – amusingly I've also had the problem with Gmail flagging the Google Groups spam reports as spam, and this halted that.

Re:Decommission of Cyrus Email by tjoff on Sunday May 07, 2017 @04:29AM (#14284452)
Same experience here, I've lost important mails that cost me money and because of that I've periodically been checking my spam folder - which is a real pain since the spam-folder isn't synced to my external mail client.

So I have to constantly login to their site only to check the spam-folder (talk about convenience). I've setup rules to try to do my best to NEVER send anything to the spam folder but even with catch-all rules gmail still think they know better than me.

Their support for real (external) mail clients is also quite bad, I don't see why so many people in tech use gmail.

Re:Decommission of Cyrus Email by int_19h on Sunday May 07, 2017 @12:22AM (#14283837)
For me, the killing feature is search. I'm used to being able to pull, say, the details of some order that I've made two years ago, based on a couple keywords that I vaguely remember.
Re:Decommission of Cyrus Email by danieldk on Sunday May 07, 2017 @01:49AM (#14284064)
That's not exclusive to Google Mail. I have never noticed any regression in search since switching to Fastmail.
Re:Decommission of Cyrus Email by int_19h on Sunday May 07, 2017 @03:08AM (#14284267)
It used to be that their search worked a lot better - but I've been using them since they were invite-only (I think it was 2004?), and haven't really looked much at other alternatives.

One other thing that I kinda sorta like in GMail is labels (as opposed to folders). As I understand, it's not a popular feature, because there's no reliable way to map it to IMAP? So anything that assumes that IMAP (rather than a custom client or web app) will be used as a mail reader, will probably only provide folders.

Re:Decommission of Cyrus Email by mavhc on Sunday May 07, 2017 @03:50AM (#14284367)
What's weird is IMAP has keywords, but they can't be used exactly like labels, no idea why IMAP hasn't been extended to include labels, seems rather easy.
Re:Decommission of Cyrus Email by brongondwana on Sunday May 07, 2017 @05:18AM (#14284589)
The hardest bit when building a client is that keywords are per-folder, so unless you store all the email in a single folder server-side, you have to run the keyword search in every folder, every time, and you can't get updates to multiple folders at once, so you have to re-run that search frequently, which hammers servers pretty hard.
Re:Decommission of Cyrus Email by raphman on Sunday May 07, 2017 @05:28AM (#14284615)
Huh, I've been extensively using custom IMAP flags/labels with Opera M2. It worked great.
Re:Decommission of Cyrus Email by int_19h on Sunday May 07, 2017 @05:39AM (#14284643)
Opera is the only non-web client that I know of that has labels. But how does it actually represent them on IMAP level? And what happens when you view the same mailbox using another client?
Re:Decommission of Cyrus Email by CPLX on Saturday May 06, 2017 @07:45PM (#14282949)
Well we did get the flipside of that, which is that there really aren't many good open platforms where you can send digital communication that you can be sure will reliably reach someone, except for maybe SMS.
Re:Decommission of Cyrus Email by gogopuppygogo on Saturday May 06, 2017 @08:02PM (#14283019)
SMS has the two faced downsides of being more expensive to use in large quantities than email alongside being highly regulated making it easy to prosecute those who choose to spam.
Re:Decommission of Cyrus Email by JohnTHaller on Saturday May 06, 2017 @09:44PM (#14283400)
Spam filtering is the killer feature for a lot of people. I'm using Rackspace hosted email on a paid basis at the moment. Any spammer that wants to can bypass the spam and manual filtering just by using UTF8 encoding on most of the message. It's been an open bug in the product for 2 years now and will likely never be fixed.
Re:Decommission of Cyrus Email by Semaphor on Sunday May 07, 2017 @12:59AM (#14283941)
> For me, the killer feature of Gmail that's hard to compete with is spam filtering.

I have absolutely no problem with Fastmail's spam filtering. Maybe Gmail changed things since I switched, around 3-4 years ago, but with both, I used to get 0-1 spam emails per month, though I haven't gotten any on FM in the last few months.

Re:Decommission of Cyrus Email by fencepost on Saturday May 06, 2017 @10:20PM (#14283497)
Spam filtering seems to be pretty solved if you're beyond the single user level. I the thing I'd value more is malware filtering in attachments.
Re:Decommission of Cyrus Email by _wmd on Saturday May 06, 2017 @08:25PM (#14283122)
One of the simplest things to do in software land is cloning an existing application, all the design work has pretty much been done for you and there is legal precedent even supporting direct rips of the look and feel of a UI (see e.g. https://en.wikipedia.org/wiki/Lotus_Development_Corp._v._Bor.... ).

While it would be quite straightforward to clone an almost pixel-perfect version of Gmail, there is something in hacker culture that makes this kind of project unattractive. I don't really understand why.

Instead what we end up with when someone starts a new mail app is some half-cocked design that makes sense for that particular user, who is always an engineer, and the result is suboptimal for pretty much anyone else.

So the project fizzles and dies before it even gets started. This happens over and over again even for larger and more visible projects like Mailpile

Re:Decommission of Cyrus Email by colechristensen on Saturday May 06, 2017 @10:20PM (#14283493)
You can clone the UI, but that's not where the work or the value of gmail is.

Gmail is _fast_. The browser doesn't reload when you navigate around and when you click on something it gets loaded very quickly. You can't just put a shiny UI in front of an IMAP server and get this sort of functionality, you have to write your own mail server with a rather intense database backing it. I have 70,000 emails spanning a decade (mostly useless) and searching is a breeze. Everything works fine and quick with a huge volume of data. When I send messages they're actually sent in a second or two, when I click next message the next message appears in a fraction of a second. The UI is the frosting on the cake. It's critical that it's very fast, but the underlying mechanics have to be very fast too, and that's not just a manner of copying pixels.

Re:Decommission of Cyrus Email by zhte415 on Saturday May 06, 2017 @11:52PM (#14283753)
Gmail is not fast.

It is fast when a huge amount of JS gets loaded and latency times are low. But with high latency? A horrible experience and if it even works I'm grateful.

It is pitifully slow when on a limited connection. And the contortionist-like procedures needed to log in from an unfamiliar IP don't make it slow, just largely non responsive.

Re:Decommission of Cyrus Email by abrowne on Saturday May 06, 2017 @10:24PM (#14283508)
> You can't just put a shiny UI in front of an IMAP server and get this sort of functionality,

Have you tried FastMail? They use Cyrus...

> you have to write your own mail server

... but I concede they (FastMail) do sponsor most of Cyrus's development.

Re:Decommission of Cyrus Email by brongondwana on Sunday May 07, 2017 @05:25AM (#14284604)
We patched Cyrus extensively to provide the cross-folder searching stuff and other non-standard stuff we needed to make our interface fast. It's all in the upstream project now (and in the 3.0 release), so anyone can use it. No standard IMAP client will speak it of course, and we strongly discourage it. We'd prefer people use JMAP, which is documented and working towards becoming a standard. It's also in 3.0 :)
Re:Decommission of Cyrus Email by andris9 on Saturday May 06, 2017 @11:53PM (#14283756)
This is exactly one of the reasons why I created my own IMAP server that uses MongoDb for mail store - to be able to access the emails fast without IMAP in between https://github.com/wildduck-email/wildduck
Re:Decommission of Cyrus Email by _wmd on Saturday May 06, 2017 @10:45PM (#14283572)
Mailpile went the custom server route, it's not exactly a turing complete problem
Re:Decommission of Cyrus Email by zakki on Saturday May 06, 2017 @10:10PM (#14283467)
Correct link: https://en.m.wikipedia.org/wiki/Lotus_Development_Corp._v._B....

I think you deleted the last "."

Re:Decommission of Cyrus Email by _wmd on Saturday May 06, 2017 @10:12PM (#14283471)
Curious, looks like an HN bug. Try this one https://en.wikipedia.org/wiki/Lotus_Development_Corp._v._Bor...
Re:Decommission of Cyrus Email by duncan_bayne on Sunday May 07, 2017 @02:57AM (#14284241)
> It's really disappointing that nothing better than gmail has come along.

That's what I thought, until I discovered Fastmail. I'm now a happy family account holder, enjoying great support from human beings, and happy that my children won't be data mined for advertising (at least, not in this particular case).

I'm not kidding about the support either. One of their developers was on Github helping out with an issue on an open source tool that, in the end, proved to be a problem with their implementation of a protocol. Fix was in Fastmail staging a week later, and in prod a few weeks after that.

Re:Decommission of Cyrus Email by bitmapbrother on Sunday May 07, 2017 @03:35AM (#14284335)
CMU uses G Suite so they do have support and their email is not "datamined"

>Unlike Google’s consumer offerings, which may show ads, we do not collect, scan or use your G Suite data for advertising purposes and do not display ads in G Suite, Education, or Government core services. We use your data to provide the G Suite services, and for system support, such as spam filtering, virus detection, spell-checking, capacity planning, traffic routing, and the ability to search for emails and files within an individual account.

Re:Decommission of Cyrus Email by duncan_bayne on Sunday May 07, 2017 @05:01AM (#14284540)
> CMU uses G Suite so they do have support and their email is not "datamined"

Sure. I was talking about their "consumer" offerings though.

Re:Decommission of Cyrus Email by jsmthrowaway on Sunday May 07, 2017 @06:35AM (#14284789)
I'd think very hard about that "for advertising purposes" hedge before claiming fully that mail is not mined. It's also not clear if "collect" and "scan" attach to that hedge as well, or if they're in totality and "use" is the only one hedged. Oxford commas have become relevant in contract law before, oddly enough.

I agree it's unlikely, I'm just saying your quote leaves several possibilities open in unkind interpretations.

Re:Decommission of Cyrus Email by tambourine_man on Saturday May 06, 2017 @08:17PM (#14283076)
I don't qualify for a 2 foot grey beard yet, but if you set up /etc/mail/access to only accept your domain, virtualusertable to your emails only, you should be set. Oh, and configure fail2ban for sendmail.

Oh yeah, and MX records take some time to update within popular services, so some gmail users might be able to reach you while others will get a user not found error. If you think about it, it makes sense, but it drove me nuts the first time.

Re:Decommission of Cyrus Email by ReligiousFlames on Sunday May 07, 2017 @01:20AM (#14283990)
Zimbra and Roundcube are the closest OSS solutions. After running an email startup, it's a very hard business to crack (people change email providers infrequently and what they're willing to pay doesn't justify investment except at massive scale.)

Zoho is sorta free and good for BYO domain-based basic web/email.

Re:Decommission of Cyrus Email by KaiserPro on Sunday May 07, 2017 @05:10AM (#14284558)
running zimbra is a massive massive faff.

custom bundled ldap server with lots of java custom goop. Not fun to support at scale

Re:Decommission of Cyrus Email by ReligiousFlames on Sunday May 07, 2017 @04:27PM (#14287453)
BTDTBTTS. That's an ignorant mischaracterization and it's not hard to do with competent staff. My team and I ran several full HA Zimbra (when owned by Vmware) deployments including major components MySQL, Lucene and Webmail (the only JVM components) OpenLDAP, Postfix, Dovecot, Nginx and various antispam plugins.
Re:Decommission of Cyrus Email by inimino on Saturday May 06, 2017 @11:55PM (#14283761)
I'm using Mail in a Box, it's not perfect but it's pretty good, and no beard required.