News for hackers. Stuff that matters.
Re:Google Apps Services Were Down by minimax on Wednesday April 17, 2013 @08:42AM (#5564075)
This is the message:

"Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience."

0900 CDT edit: works for me now.

Re:Google Apps Services Were Down by viame on Wednesday April 17, 2013 @09:20AM (#5564284)
Getting the same message. However, this is happening to some of my logins only. Same with e-mails, some of them are working fine and some of them are timing out or asking for passwords. For a second I thought someone hacked my e-mail. :>
Re:Google Apps Services Were Down by g-garron on Wednesday April 17, 2013 @09:31AM (#5564344)
Me too, I was really scared. Now I know is Google's problem I am breading again.
Re:Google Apps Services Were Down by laumars on Wednesday April 17, 2013 @09:39AM (#5564386)
> Me too, I was really scared. Now I know is Google's problem I am breading again.

I hate it when Google get in the way of my sandwich making ;-)

Re:Google Apps Services Were Down by Samuel_Michon on Wednesday April 17, 2013 @08:45PM (#5568307)
Given the naming scheme of Android releases, it seems like Google would let you eat cake instead.
Re:Google Apps Services Were Down by kaolinite on Wednesday April 17, 2013 @09:59AM (#5564496)
Same.. I was the only person in the company unable to access emails. Was scared I was about to be called into a boardroom ;-)
Re:Google Apps Services Were Down by saurik on Wednesday April 17, 2013 @04:45PM (#5567083)
That error message seems to just be Google's default 500 error message page; even when it is an API request that fails, even if the API is specified to return errors as some kind of JSON or XML, you get that HTML page in response. (I know this as Google's OpenID+OAuth pipeline has been failing occassionally for the last month, something I've been trying to get them to fix on the Google Federated Login API mailing list.)
Re:Google Apps Services Were Down by infoman on Wednesday April 17, 2013 @04:52PM (#5567142)
so many developers hacking on google glass apps?
Re:Google Apps Services Were Down by TranceMan on Wednesday April 17, 2013 @09:11AM (#5564228)
Google Apps status page: http://www.google.com/appsstatus
Re:Google Apps Services Were Down by AndrewDucker on Wednesday April 17, 2013 @10:26AM (#5564631)
The Register reports that GMail went down exactly a year ago:

http://www.theregister.co.uk/2012/04/18/google_gmail_outage/

Re:Google Apps Services Were Down by saurik on Wednesday April 17, 2013 @05:50PM (#5567531)
Huffington Post reports that Gmail also went offline on April 29th of 2010.

http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gm...

Google reports that Gmail also went offline on April 24th of 2009.

http://googleblog.blogspot.com/2009/02/current-gmail-outage....

(Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this would be an epic pattern of "don't trust Gmail during the end of April" ;P.)

http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-...

Re:Google Apps Services Were Down by ndespres on Wednesday April 17, 2013 @09:27AM (#5564322)
I'm a Google Apps reseller and thought, oh no, I should call all my clients and let them know about this outage before they call me! Where can I get a full customer list.. the admin panel.. oh.
Re:Google Apps Services Were Down by stedaniels on Wednesday April 17, 2013 @09:29AM (#5564335)
Your only copy of your customers, is in a third party reseller control panel?
Re:Google Apps Services Were Down by ndespres on Wednesday April 17, 2013 @09:43AM (#5564409)
Of course not. Just the fastest way to get them! Fortunately, Confluence, Autotask, and my memory are all still intact!
Re:Google Apps Services Were Down by mathattack on Wednesday April 17, 2013 @10:23AM (#5564622)
"and my memory"

When all else fails, this is the true old school solution. :-)

Re:Google Apps Services Were Down by lewispb on Wednesday April 17, 2013 @09:29AM (#5564334)
The phone calls from my clients have started to come in already...!
Re:Google Apps Services Were Down by jqueryin on Wednesday April 17, 2013 @09:31AM (#5564342)
There are API endpoints for that. Not sure if they're affected.
Re:Google Apps Services Were Down by josephb on Wednesday April 17, 2013 @08:46AM (#5564099)
Nearly 30 minutes since it started and the status page is it's customary blank screen.

http://imgur.com/OLyGf0O

Almost as bad as AWS at getting notices up.

Re:Google Apps Services Were Down by guilhermetk on Wednesday April 17, 2013 @08:48AM (#5564116)
Now it is showing Service disruption for Admin control panel / API

http://www.google.com/appsstatus#hl=en&v=status&ts=1...

Re:Google Apps Services Were Down by WimLeers on Wednesday April 17, 2013 @08:48AM (#5564114)
Odd, "Admin control panel/API" is marked as disrupted for me.
Re:Google Apps Services Were Down by josephb on Wednesday April 17, 2013 @08:50AM (#5564127)
Yeah it is now, only took 30 minutes :-)

The outage is affecting IMAP/POP/HTTP for email plus Gchat, the admin control panel/API is probably the least impact.

Re:Google Apps Services Were Down by misnome on Wednesday April 17, 2013 @09:39AM (#5564385)
Well, their email services are still working, because I got a message about our trial ending today and that we need to start paying. I hope this comes up before we get automatically cut off..
Re:Google Apps Services Were Down by JosephRedfern on Wednesday April 17, 2013 @08:53AM (#5564144)
Getting IMAP rejections, and Google Docs doesn't seem to be working: http://cl.ly/image/3W2e0X1O1i0o
Re:Google Apps Services Were Down by wodow on Wednesday April 17, 2013 @08:56AM (#5564167)
smtp.googlemail.com also failing

(edit: intermittently)

Re:Google Apps Services Were Down by fish2000 on Wednesday April 17, 2013 @09:48AM (#5564439)
Crazy – my IMAP client (Apple Mail) has no problem until it tries to send via SMTP… when it chokes.

Also, no working gmail for all my accounts except my @gmail.com address – at least fail consistently, you Googlers. Blech!

Re:Google Apps Services Were Down by kyrra on Wednesday April 17, 2013 @09:01AM (#5564191)
Oddly enough, gmail itself works on my Google Apps domain. But accessing my account page does not work. As well, I can't log into Gchat using one of the generated passwords (says password rejected). I'm guessing this is the same reason IMAP is failing for others.
Re:Google Apps Services Were Down by Kristories on Wednesday April 17, 2013 @08:46AM (#5564098)
Re:Google Apps Services Were Down by ac132 on Wednesday April 17, 2013 @08:56AM (#5564164)
At least Google Reader still works :-)
Re:Google Apps Services Were Down by logn on Wednesday April 17, 2013 @03:31PM (#5566547)
I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?
Re:Google Apps Services Were Down by bdonlan on Wednesday April 17, 2013 @10:11PM (#5568579)
Search is (mostly) stateless. They can copy the search database to hundreds of DCs around the world, and each can operate independently - so if one of them has a problem, it's not likely to affect the others (unless it's bad data, but that's why you shouldn't deploy index updates to all DCs simultaneously). There's some personalization, sure, but you can just fall back to non-personalized search if that breaks.

Things like gmail, on the other hand, are inherently stateful. When you log into gmail, you have to eventually connect to one system that maintains your mailbox. Sure, there might be replication - but the replicas are all talking to each other. It's surprisingly easy to have a cascade failure in a system like this, where one of the replicas going down triggers (directly or indirectly) all the others failing as well. Or you can have some bad data that gets replicated out, and then proceeds to confuse everything that's looking at it - unlike search, you have to replicate that data immediately, and don't get to enjoy the benefits of a staged deployment.

This also explains why not all users were affected - I'd guess that their system is divided into some number of shards, and users are assigned to a particular shard. That 0.07% of users affected probably represents a single unhealthy shard.

Re:Google Apps Services Were Down by therealarmen on Wednesday April 17, 2013 @04:48PM (#5567106)
Contributing the vast majority of revenue for the company has something to do with it.
Re:Google Apps Services Were Down by freehunter on Wednesday April 17, 2013 @09:23AM (#5564305)
I'm wondering if this is related to the new update they did to the admin control panel.

http://arstechnica.com/information-technology/2013/04/make-g...

Re:Google Apps Services Were Down by corresation on Wednesday April 17, 2013 @09:52AM (#5564461)
There is a massively linked Boston bomber Google Docs spreadsheet making the rounds -- the last time a viral Google doc saw that sort of exposure, the service went down.
Re:Google Apps Services Were Down by appleflaxen on Wednesday April 17, 2013 @08:44AM (#5564088)
Since you always worry you are going crazy when a high-uptime site is down: mine too

FWIW: I was able to log in by going to www.gmail.com and logging in suing [email protected] as the username.

Re:Google Apps Services Were Down by bkanber on Wednesday April 17, 2013 @09:06AM (#5564211)
Tried this -- didn't work for me unfortunately.
Re:Google Apps Services Were Down by terryjsmith on Wednesday April 17, 2013 @08:43AM (#5564078)
I'm getting IMAP logins rejected as well.
Re:Google Apps Services Were Down by runemadsen on Wednesday April 17, 2013 @08:43AM (#5564082)
I'm also getting errors when trying to retrieve Gmail messages in IOS Mail. "Wrong username or password", but it works in a browser.
Re:Google Apps Services Were Down by js2 on Wednesday April 17, 2013 @08:46AM (#5564097)
It's intermittent for me. I first noticed trouble when the Android gmail app said it couldn't connect for one of my accounts. I cannot login to my apps domain control panel right now, but I can access its email via both IMAP and the web interface. However, another account on that same domain cannot access via IMAP.
Re:Google Apps Services Were Down by Leszek on Wednesday April 17, 2013 @09:52AM (#5564465)
This explains the SMTP problems I was having, I was worried someone had hacked my account or something...
Re:Google Apps Services Were Down by yaskyj on Wednesday April 17, 2013 @08:53AM (#5564142)
It's interesting that I kind find news for this on google itself, but Hacker News already has two posts.
Re:Google Apps Services Were Down by glennos on Wednesday April 17, 2013 @08:52AM (#5564132)
The dashboard hasn't caught up. Admin Control Panel is one thing, but GMail/Apps has a huge impact!
Re:Google Apps Services Were Down by dotbill on Wednesday April 17, 2013 @10:06AM (#5564531)
As I said in the other google apps thread on hackernews..

I thought I'd give them a call to find out whats going on, however to call them I need my support/admin pin - which is only accessible from the admin portal and without it they won't answer the phone. Nice.

Re:Google Apps Services Were Down by silasb on Wednesday April 17, 2013 @08:59AM (#5564181)
Appears Groups is down also.
Re:Google Apps Services Were Down by brokentone on Wednesday April 17, 2013 @11:29AM (#5565003)
Still getting errors sending transactional messages to users on gmail

Apr 17 10:23:30 postfix/smtp: connect to gmail-smtp-in.l.google.com[2607:f8b0:4002:c02::1a]:25: Network is unreachable

Re:Google Apps Services Were Down by mattvot on Wednesday April 17, 2013 @08:52AM (#5564134)
I did wonder about this. My phone said my username or password was wrong for my email, got a bit worried, so I tried with their webmail and it worked there.
Re:Google Apps Services Were Down by theg2 on Wednesday April 17, 2013 @08:47AM (#5564108)
Was having problems with play developers console yesterday, suddenly it was a 404. Seems like they're planning something in my opinion.
Re:Google Apps Services Were Down by X-Istence on Wednesday April 17, 2013 @09:12AM (#5564235)
That explains why I can't get my email =).
Re:Google Apps Services Were Down by bdg on Wednesday April 17, 2013 @08:45AM (#5564094)
This is happening for gmail for some accounts, but mine seem to be running. Seems the outage is per user account.
Re:Google Apps Services Were Down by jlengrand on Wednesday April 17, 2013 @09:44AM (#5564418)
I got a 500 when I try to access our Drive too. It's been an hour now. Using the business google services.
Re:Google Apps Services Were Down by delinka on Wednesday April 17, 2013 @08:50AM (#5564125)
I also can't authorize an app to have access to a Gmail account. Google's auth page presents an error.
Re:Google Apps Services Were Down by traeregan on Wednesday April 17, 2013 @08:54AM (#5564151)
Same here. Intermittent though. I'm like 2 for 5.
Re:Google Apps Services Were Down by arcosdev on Wednesday April 17, 2013 @08:57AM (#5564173)
Faaack. I thought I was just having a local connection issue. Same problem here.
Re:Google Apps Services Were Down by digitalpbk on Wednesday April 17, 2013 @09:42AM (#5564401)
digitalpbk$ nslookup smtp.google.com Server: 8.8.8.8 Address: 8.8.8.8#53

server can't find smtp.google.com: NXDOMAIN

SMTP servers are down for me. Anyone else?

Re:Google Apps Services Were Down by brokentone on Wednesday April 17, 2013 @09:53AM (#5564471)
NXDOMAIN wouldn't be server down, that's no DNS record. Still won't work, but different (much bigger) issue.
Re:Google Apps Services Were Down by tekacs on Wednesday April 17, 2013 @10:02AM (#5564507)
I'm pretty sure the 'bigger' issue here is that it's smtp.gmail.com - smtp.google.com is, I believe, always down on account of not existing. :P
Re:Google Apps Services Were Down by deno on Wednesday April 17, 2013 @10:52AM (#5564797)
Works fine for me:

    > dig +short @8.8.8.8 smtp.gmail.com
    gmail-smtp-msa.l.google.com.
    74.125.136.108
    74.125.136.109
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nsl...
Re:Google Apps Services Were Down by pyre on Wednesday April 17, 2013 @10:17AM (#5564586)
Or: smtp.googlemail.com
Re:Google Apps Services Were Down by duncans on Wednesday April 17, 2013 @10:16AM (#5564581)
try `nslookup -querytype=MX google.com`
Re:Google Apps Services Were Down by bkanber on Wednesday April 17, 2013 @08:52AM (#5564139)
Down here. Additionally, my iCal won't sync with goocal.
Re:Google Apps Services Were Down by meerita on Wednesday April 17, 2013 @10:04AM (#5564513)
I cannot access my email from Sparrow app too.
Re:Google Apps Services Were Down by frankydp on Wednesday April 17, 2013 @08:52AM (#5564138)
Webmaster tools also down with auth failure.
Re:Google Apps Services Were Down by jschuur on Wednesday April 17, 2013 @08:48AM (#5564111)
Google Drive is down for me on Apps too.
Re:Google Apps Services Were Down by varun1308 on Wednesday April 17, 2013 @08:55AM (#5564160)
Yeah.. my pop mail stopped working too
Re:Google Apps Services Were Down by calleskonto on Wednesday April 17, 2013 @09:14AM (#5564239)
The provision api is still fine doh.
Re:Google Apps Services Were Down by ckdarby on Wednesday April 17, 2013 @08:19PM (#5568235)
Why do people report this kind of stuff on hacker news. I am sure everyone knows how to google for a site's status page.
Re:Google Apps Services Were Down by EvanAnderson on Wednesday April 17, 2013 @09:45AM (#5564422)
SAML logons are failing, too.
Re:Google Apps Services Were Down by paulmolluzzo on Wednesday April 17, 2013 @09:28AM (#5564325)
It's back.
Re:Google Apps Services Were Down by level09 on Wednesday April 17, 2013 @09:43AM (#5564408)
not only that, seems like pop3 is also facing issues ..
Re:Google Apps Services Were Down by abhi_21 on Wednesday April 17, 2013 @09:45AM (#5564421)
Now its working..
Re:Google Apps Services Were Down by Selfcommit on Wednesday April 17, 2013 @11:37AM (#5565049)
And We're up.
Re:Google Apps Services Were Down by eggnet on Wednesday April 17, 2013 @08:43AM (#5564080)
down for me too