Re:Google Apps Services Were Down
by minimax on Wednesday April 17, 2013 @08:42AM
(#5564075)
This is the message:
"Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience."
Re:Google Apps Services Were Down
by viame on Wednesday April 17, 2013 @09:20AM
(#5564284)
Getting the same message. However, this is happening to some of my logins only. Same with e-mails, some of them are working fine and some of them are timing out or asking for passwords. For a second I thought someone hacked my e-mail. :>
Re:Google Apps Services Were Down
by saurik on Wednesday April 17, 2013 @04:45PM
(#5567083)
That error message seems to just be Google's default 500 error message page; even when it is an API request that fails, even if the API is specified to return errors as some kind of JSON or XML, you get that HTML page in response. (I know this as Google's OpenID+OAuth pipeline has been failing occassionally for the last month, something I've been trying to get them to fix on the Google Federated Login API mailing list.)
(Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this would be an epic pattern of "don't trust Gmail during the end of April" ;P.)
Re:Google Apps Services Were Down
by ndespres on Wednesday April 17, 2013 @09:27AM
(#5564322)
I'm a Google Apps reseller and thought, oh no, I should call all my clients and let them know about this outage before they call me! Where can I get a full customer list.. the admin panel.. oh.
Re:Google Apps Services Were Down
by misnome on Wednesday April 17, 2013 @09:39AM
(#5564385)
Well, their email services are still working, because I got a message about our trial ending today and that we need to start paying. I hope this comes up before we get automatically cut off..
Re:Google Apps Services Were Down
by kyrra on Wednesday April 17, 2013 @09:01AM
(#5564191)
Oddly enough, gmail itself works on my Google Apps domain. But accessing my account page does not work. As well, I can't log into Gchat using one of the generated passwords (says password rejected). I'm guessing this is the same reason IMAP is failing for others.
Re:Google Apps Services Were Down
by logn on Wednesday April 17, 2013 @03:31PM
(#5566547)
I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?
Re:Google Apps Services Were Down
by bdonlan on Wednesday April 17, 2013 @10:11PM
(#5568579)
Search is (mostly) stateless. They can copy the search database to hundreds of DCs around the world, and each can operate independently - so if one of them has a problem, it's not likely to affect the others (unless it's bad data, but that's why you shouldn't deploy index updates to all DCs simultaneously). There's some personalization, sure, but you can just fall back to non-personalized search if that breaks.
Things like gmail, on the other hand, are inherently stateful. When you log into gmail, you have to eventually connect to one system that maintains your mailbox. Sure, there might be replication - but the replicas are all talking to each other. It's surprisingly easy to have a cascade failure in a system like this, where one of the replicas going down triggers (directly or indirectly) all the others failing as well. Or you can have some bad data that gets replicated out, and then proceeds to confuse everything that's looking at it - unlike search, you have to replicate that data immediately, and don't get to enjoy the benefits of a staged deployment.
This also explains why not all users were affected - I'd guess that their system is divided into some number of shards, and users are assigned to a particular shard. That 0.07% of users affected probably represents a single unhealthy shard.
Re:Google Apps Services Were Down
by corresation on Wednesday April 17, 2013 @09:52AM
(#5564461)
There is a massively linked Boston bomber Google Docs spreadsheet making the rounds -- the last time a viral Google doc saw that sort of exposure, the service went down.
Re:Google Apps Services Were Down
by js2 on Wednesday April 17, 2013 @08:46AM
(#5564097)
It's intermittent for me. I first noticed trouble when the Android gmail app said it couldn't connect for one of my accounts. I cannot login to my apps domain control panel right now, but I can access its email via both IMAP and the web interface. However, another account on that same domain cannot access via IMAP.
Re:Google Apps Services Were Down
by dotbill on Wednesday April 17, 2013 @10:06AM
(#5564531)
As I said in the other google apps thread on hackernews..
I thought I'd give them a call to find out whats going on, however to call them I need my support/admin pin - which is only accessible from the admin portal and without it they won't answer the phone. Nice.
Re:Google Apps Services Were Down
by mattvot on Wednesday April 17, 2013 @08:52AM
(#5564134)
I did wonder about this. My phone said my username or password was wrong for my email, got a bit worried, so I tried with their webmail and it worked there.
"Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience."
0900 CDT edit: works for me now.
I hate it when Google get in the way of my sandwich making ;-)
http://www.theregister.co.uk/2012/04/18/google_gmail_outage/
http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gm...
Google reports that Gmail also went offline on April 24th of 2009.
http://googleblog.blogspot.com/2009/02/current-gmail-outage....
(Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this would be an epic pattern of "don't trust Gmail during the end of April" ;P.)
http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-...
When all else fails, this is the true old school solution. :-)
http://imgur.com/OLyGf0O
Almost as bad as AWS at getting notices up.
http://www.google.com/appsstatus#hl=en&v=status&ts=1...
The outage is affecting IMAP/POP/HTTP for email plus Gchat, the admin control panel/API is probably the least impact.
(edit: intermittently)
Also, no working gmail for all my accounts except my @gmail.com address – at least fail consistently, you Googlers. Blech!
Service disruption
http://www.google.com/appsstatus#hl=en&v=issue&ts=13...
Things like gmail, on the other hand, are inherently stateful. When you log into gmail, you have to eventually connect to one system that maintains your mailbox. Sure, there might be replication - but the replicas are all talking to each other. It's surprisingly easy to have a cascade failure in a system like this, where one of the replicas going down triggers (directly or indirectly) all the others failing as well. Or you can have some bad data that gets replicated out, and then proceeds to confuse everything that's looking at it - unlike search, you have to replicate that data immediately, and don't get to enjoy the benefits of a staged deployment.
This also explains why not all users were affected - I'd guess that their system is divided into some number of shards, and users are assigned to a particular shard. That 0.07% of users affected probably represents a single unhealthy shard.
http://arstechnica.com/information-technology/2013/04/make-g...
FWIW: I was able to log in by going to www.gmail.com and logging in suing [email protected] as the username.
I thought I'd give them a call to find out whats going on, however to call them I need my support/admin pin - which is only accessible from the admin portal and without it they won't answer the phone. Nice.
Apr 17 10:23:30 postfix/smtp: connect to gmail-smtp-in.l.google.com[2607:f8b0:4002:c02::1a]:25: Network is unreachable
server can't find smtp.google.com: NXDOMAIN
SMTP servers are down for me. Anyone else?