News for hackers. Stuff that matters.
Re:More On Gmail’s Delivery Delays by jrochkind1 on Tuesday September 24, 2013 @12:03PM (#6438718)
A ~2.5 hour delay that effected ~1% of messages -- I've never had a mail provider so perfect that a delay of a couple hours didn't very occasionally happen.

As far as their telling us, the failure didn't cascade causing their queues to fill up and other horrible things to happen. As far as we know, no mail got bounced, no mail got dropped on the floor and lost permanently, etc.

A 2.6 second delay isn't even worth mentioning, how could one even tell the difference between a 2.6 second delay due to a problem on google's side, or routine problems in the email network outside of google that could easily lead to a second delay or whatever.

Did people really get upset about this? Was it worse than it seems because of details they are ommitting? Are they preemptively advertising how well they dealt with this barely-a-problem-noticeable-to-users problem, because the message is really "see, look how damn stable gmail is, our worst problem ever is barely noticeable"?

Or, what?

Maybe those 1.5% of messages were all to the same users, so some people had no delay, and some people had ALL their mail delayed a couple hours? That would make it more noticeable.

But in general, occasional (like once a year or less) couple-hour mail queue delays should probably be expected from any mail provider, no?

Re:More On Gmail’s Delivery Delays by nolok on Tuesday September 24, 2013 @12:19PM (#6438819)
We use google apps for business and a fair number of our account were "heavily" affected, with mail taking upward of half an hour to deliver.

Which is still really fast, especially when you know how mail work and remember the "old days". I guess people got used to email being an instantaneous form of communication, which says a lot about how great it is working most of the time, given that instantaneous is not in email's job description.

Re:More On Gmail’s Delivery Delays by jlgaddis on Tuesday September 24, 2013 @01:41PM (#6439446)
> Which is still really fast, especially when you know how mail work and remember the "old days".

Mmm, dialing in to QuantumLink at 300 baud... ahh, the good^Wold days.

Re:More On Gmail’s Delivery Delays by susi22 on Tuesday September 24, 2013 @12:58PM (#6439103)
We use email communication for our help ticket systems and all our users were affected (~20k). At

9am it was ~10min delay.

10am about 21min

noon about 1h

3pm emails with ~3h delay arrived.

Then it got better. We werent' able to help our users and they of course blamed that we responded slowly. Actually a bounce message would've been preferable since then they could've used a phone or stopped by.

You don't really expect your company @example.org to take 3h. Either bounce withing ~3-4min or you expect it to be delivered by that time.

Re:More On Gmail’s Delivery Delays by jessaustin on Tuesday September 24, 2013 @01:44PM (#6439465)
We use email communication for our help ticket systems and all our users were affected (~20k).

After this event are you considering a different architecture?

Re:More On Gmail’s Delivery Delays by gardarh on Tuesday September 24, 2013 @04:35PM (#6440579)
We noticed similar delays at our office (a lot fewer people though). Somehow all the test email we sent seem to have ended in the 1.5%. Odd.
Re:More On Gmail’s Delivery Delays by JacobJans on Tuesday September 24, 2013 @01:31PM (#6439377)
My business sends a lot of emails. Yesterday morning we were scheduled to send an email to ~8,000 of our subscribers. I sent an test email to a test list, and it did not go through. This test email went to 4 different Gmail address. None of them received it. Around 40 minutes later, one of the Gmail accounts finally received the message. At this point I sent another email to the test list. Still nothing. I tried emailing from a Yahoo account, to Gmail, and it went through. However, for over 3 hours I couldn't send an email from my server to Gmail. Emails from other servers were able to go through.

The problem does not appear to be with individual user accounts, but from Google's ability to receive emails from certain parts of the internet universe.

The result was lost time and lost money on my end. I imagine this had very little impact on the experience of individual users -- but for businesses that depend on getting emails to their customers, this had a much bigger impact.

My experience was this:

1. This problem was not a 2.6 second delay. It was a multi-hour delay.

2. The problem was not with 1.5% of Gmail users, but with Google's ability to receive messages from other parts of the internet. (A major pipe was down, according to the OP.)

3. No Gmail user had all their incoming mail delayed for a couple hours -- Certain senders to Gmail had all of their mail delayed for several hours.

Re:More On Gmail’s Delivery Delays by mhurron on Tuesday September 24, 2013 @03:37PM (#6440201)
> This problem was not a 2.6 second delay. It was a multi-hour delay.

Technically this is not a problem. SMTP does not guarantee instantaneous delivery, just that if a SMTP server accepts the message it will be delivered.

In reality, this 'delay' could be perfectly normal and is just fine according to spec.

Re:More On Gmail’s Delivery Delays by dangrossman on Tuesday September 24, 2013 @12:14PM (#6438790)
> But in general, occasional (like once a year or less) couple-hour mail queue delays should probably be expected from any mail provider, no?

I've hosted a dozen or so mailboxes at Rackspace Mail (formerly Mailtrust) for several years now. I have experienced zero delayed delivery or service failures of any kind in all those years.

For $2/mo/mailbox ($10/mo minimum total), you get a 100% uptime SLA, 25GB per mailbox, IMAP-push/POP3, good spam filtering and 24/7/365 support. That's 100% uptime or you get paid, and a real person will pick up the phone if you have a problem at 3AM on Christmas, for $2.

Re:More On Gmail’s Delivery Delays by nolok on Tuesday September 24, 2013 @12:24PM (#6438850)
Getting an email server to work in 2013 is relatively easy. Fixing it quickly without losing anything when a problem arises and things pile up is not.

Having your worst damage in case of critical failure being only 2.5 hours of delay, on a very large scale architecture ? That's genuinely great work. 2.5 hours is not even a failure in email terms.

Re:More On Gmail’s Delivery Delays by Erwin on Tuesday September 24, 2013 @05:26PM (#6440875)
The SLA isn't really great (but that's not exclusive to RS). For RS managed hosting SLA 30 minutes of network downtime means 5% of monthly fee back.

So if this counted as downtime (and that's unlikely) and your email was down for 3 hours during your business hours, and if the mail SLA is the same as managed hosting's network SLA you'd get back 60 cents per user.

Anyway, Google Apps also has a SLA which would credit 10% of the monthly for less than 99.9% -- but again, that's uptime so "some users not getting email on time" is not going to count.

Re:More On Gmail’s Delivery Delays by swalkergibson on Tuesday September 24, 2013 @12:09PM (#6438748)
This. Absolutely this. Considering hosting your own service involves paying a competent system administrator at least $75K/year in addition to buying hardware and network and power, $5/user/month is more than reasonable for such an extremely high level of service. If some mission critical email was delayed for a little over an hour, pick up the phone. This is the very definition of a mountain out of a molehill.
Re:More On Gmail’s Delivery Delays by jlgaddis on Tuesday September 24, 2013 @01:34PM (#6439403)
While "1.5% of messages" doesn't sound like that much, it probably added up to millions of e-mails over that ~2.5 hour period.

(Also: it's probably not feasible to do for everyone at your companies, but offlineimap[0] is a great tool for ensuring that you always have a local copy of your mailbox.)

[0]: http://offlineimap.org/

Re:More On Gmail’s Delivery Delays by rory096 on Tuesday September 24, 2013 @12:25PM (#6438858)
>Maybe those 1.5% of messages were all to the same users, so some people had no delay, and some people had ALL their mail delayed a couple hours?

Probably this. Gmail was just about unusable for me yesterday.

Not that I'm complaining, for exactly your reasons.

Re:More On Gmail’s Delivery Delays by eitally on Tuesday September 24, 2013 @12:26PM (#6438866)
Definitely this. I manage a 20,000+ GApps domain and a lot of folks weren't affected at all, but some people (myself included) only received a trickle of messages all day, with a flood coming in late afternoon.
Re:More On Gmail’s Delivery Delays by edgesrazor on Tuesday September 24, 2013 @01:18PM (#6439271)
I'm seeing quite a few comments across the web from people saying they were in the 1.5%, which makes me wonder how accurate that number is. Then again, Gmail has a LOT of users, so that could be a very accurate number.

We use Google Apps for Business and were experiencing delays of several hours, and I did receive some bounces when sending to others within my organization. Strangely, most of this happened when using Outlook with Google Apps Sync - not when using the web interface.

Still, we've been very happy with the service - things like this happen every once in a while. We made do until it was straightened out.

Re:More On Gmail’s Delivery Delays by jlgaddis on Tuesday September 24, 2013 @01:44PM (#6439458)
How many of the other 98.5% weren't even aware that anything happened and, thus, haven't complained about it? My mother probably doesn't even have a clue that there was an issue.
Re:More On Gmail’s Delivery Delays by thedufer on Tuesday September 24, 2013 @03:54PM (#6440288)
No one was in the 1.5%. It was 1.5% of messages. For example, maybe 1/4 of my incoming emails had delays. If that was true of everyone who had delays, now you're looking at 6% of people affected.
Re:More On Gmail’s Delivery Delays by pfortuny on Tuesday September 24, 2013 @01:12PM (#6439216)
Hi,

I guess the people who got upset are those whose email got delayed hours (that tiny percentage of users which amounts to some dozens).

"A 2.6sec delay" is just the median (as per the OP), that leads nowhere if you do not know the mean or other statistics. Which was the maximum delay?

Just trying to clarify that the fact that you are not upset does not mean the 50% people above 2.6 secs should not.