HN is my goto status page when these things happen, never failed to provide up-to-date reliable information.

Seems like everyone going on HN now. The site is so slow

Interesting cascade effect of sites going down! I wonder where people will go next to check if HN is down?

Also wondering if this is perhaps the fastest upvoted HN post ever? 8 mins -> ~350 votes, 15 mins -> ~750 votes. I wonder if @dang could chime in with some stats?

Update: looks like it hit 1000 upvotes in ~25 mins!

Update: 1500 in ~40 mins

Update: 2000 in ~1 hour 20 mins (used the HN API for the timestamp)

The recent youtube-dl takedown had a similar number of votes but still slightly slower, I think. I think it was >500 in 30min and >1000 in 60min if I remember correctly.

Some politics topics have shot up pretty quickly in the past but user and/or mod flags send them back to page 2 about as quickly.

Funny this post is already #7. Are we seeing many reports again?

Stats right now: 1985 points | 1 hour ago | 597 comments

Maybe, could have just seen a dropoff in upvotes once the issue was resolved also

reddit, nanog, 4chan, twitter — pick your poison

"poison" is a particularly apt description for all of the above

> I wonder where people will go next to check if HN is down?

Probably: https://downforeveryoneorjustme.com/

HN was having problem showing anything for the past 10 min or so. I was going to ask if HN was down, then I saw this.

HN uses Firebase, maybe that is the reason.

Does this site also use Firebase, or do they only feed Firebase's Database so that other developers can build upon it?

Any source for that? Last information I have is that it's written in Arc Lisp and uses files rather than a database for storage.

There is a public API on Firebase, but AFAIR it's just a mirror rather than the main storage.

TIL. What for?

The very least public APIs are exposed via firebase.

https://github.com/HackerNews/API

Accessing the site anonymously fixes it (logged sessions are not chaced)

I, and probably many others here are like a status page for friends and family too... I had my wife thinking that the internet was broken, and tried tethering to phone and things and still didn't work, then showed me and I saw the status code errors and was like "it's actually parts of Google that are down".

I love (and am deeply scared by) the dependence of Google and the confusion of it with the entire internet.

It gets even worse: https://twitter.com/joemfbrown/status/1338452107419148290

>I’m sitting here in the dark in my toddler’s room because the light is controlled by @Google Home. Rethinking... a lot right now.

Some people are compiling more relevant events: https://twitter.com/internetofshit

I still like my physical switch on the wall to turn the light on and off very much.. for me it's hard to beat. I can even turn it on/off semi-remotely with e.g. a tennis ball.

If my Alexa fails, then I can also just turn the light off with the switch and when flipped back "on" again, then it will be on.

Fallback to 'classical mode' works for me.

If your home automation system requires a network connection to work then something is very wrong.

Right? Home automation is perfectly fine and mught even expose a secure authenticated API to the internet so you can say check and adjust the temperature remotely, but it shoukd never ever go down for lical use if the Internet connectivity or the remote service backend goes down.

Home Assistant rulez with non-cloud sensors. I never by sensors, cameras, switches which require connection to an external host. But I'm sure that most people will do it even with such outages too.

For many others it seems: https://i.ibb.co/ns8DgGq/Screenshot-2020-12-14-at-12-10-17.p...

Google should write an AI application that checks on HN if their systems are working and display the result live on their status page. That would be more reliable than the current system they have in place (which is obviously not working at all).

It would probably just end up banning Googles own account and leave them with no human escalation option.

won't help when the ai service itself is down.

There are other sites that can check for you

https://downdetector.com/ is remarkably good at catching this too.

But funnily enough, a lot of the votes come from traffic that searches for "is ____ down?" on Google. XD

https://downdetector.com/search/?q=google shows all the Google services and seems like everything other than search is reporting errors

Even search seems to have issues. It can't seem to find rare pages at all, and ranking seems subpar for the past 15 minutes or so.

I think that's normal. ;)

Amusingly, downdetector.com is now down. Gotta love that cascade effect :-)

Found this relatively recently, seems to do a good job.

Do love how more consumer services (ISPs &c) always have some report of being down somewhere but its means nothing unless there's a big spike.

What's the point of having a status page when it can't reliably tell you the factual status?

Helps Google meet a 99.999% SLA!

Not really since now it's up, and the status page still says it's down.

right after I shamed them on twitter. I take responsibility despite it unlikely having any impact.

Especially one that breaks the back button.

Customer confidence. Same as how everyone reports they have 99.99% uptime.

Then their status page is not unlike like target reporting in the USSR: completely fictional.

From personal experience and other Ops peoples stories, I'd assume that could be the case for many status pages ;)

It's red already.

No -- it needs to be red when it's having the outage for people to have confidence in it as an indicator. The reality is Google have no real incentive to provide an actual external status page that is accurate -- to do so is an admission of not upholding SLAs. These status pages are updated retrospectively. Use a third party one like DownDetector.

It is all green if you do not need to be logged in.

If you are logged in, the page crashes with an error.

You can still browse all services from Incognito (which for some is not an option).

Are you saying the Google status page is correct then? Being logged in is part of being green I’d think.

Also, you can’t use many parts of Gmail, Drive, Photos, etc, without being logged in.

I think it just means the status page itself is not logged in. Apparently it should be...

Try YouTube, which is totally broken.

But I guess it's technically not part of /appsstatus

Youtube is down for me if I am logged in. I can watch videos through duckduckgo search or an icognito window.

YT seems to be working in Incognito / Private mode.

Ah, thanks!

Kinda weird it would totally break if the auth failed, unlike other services like Search.

Youtube Tv too

youtube is working in incognito mode

Yeah for me too: youtube and gmail works in incognito mode

> gmail works in incognito mode

I mean, that must be a generous definition of "works"! :)

Yeah xD

Is HN having issues too?

drive too.

Google made everyone get an account trying to force Google+ on to everyone. They don't get to pretend like that didn't happen by not factoring it into their status.

This is not true. i am logged in to gmail and getting 500 err

edit:parent updated comment

That's what the GP is saying.

the parent comment to mine at time of writing was just:

> It is all green if you do not need to be log in.

Giving the impression if you were logged in/didn't need to log in, that everything would be ok.

I apologize for my ambiguous phrasing, and I feel bad that you were downvoted (I reupvoted you, but that’s only one vote).

I wrote too fast because I thought it could help people work around the problem.

Oh, I see. Looks like their auth (state) servers are down.

It reflects the reality for them.

The small print says: The problem with Gmail should be resolved for the vast majority of affected users. We will continue to work towards restoring service for the remaining affected users...

At Google scale, "remaining affected users" probably number in tens of millions. Sucks to be one of them, tho.

But hey, it happens. As a SaaS maintainer, I can sympathise with any SREs involved in handling this, and know that no service can be up 100% of the time.

Everything went red just now.

Someone here in hn said at Amazon AWS they are required direct CTO approvals to reflect negative realities. Maybe that’s the case for anyone these days.

Not true. Only for written things like postmortems. Automated systems need to be vetted but can update things like status pages on their own.

The issue isn't "negative" realities, it is saying something while investigating that might break contracts only to find later that it wasn't true.

Mistakes happen... that page is probably still correct 99,9999% of the time!

/s

These uptime numbers cloud providers gives out can't be nowhere near actual uptime.

Solid red now. Everything down

Definitely i agree with you. Although the statuses are updated later, it may cause many problems if it is not instantaneous.

It's correct for me now

Pretty funny. I would normally assume they are connected to some testing unit that real users interface with. Vs internal APIs or however Google is doing this with them still being green

Everything red now: clean sweep

It started showing red now

Now it's all red

everything is red now.

its not green anymore

I don't think it's difficult, I think they just lie.

Monitoring is very simple, I even learned this from a document released by the Google devops team many years ago.

Always alert from the end user perspective. So in other words have an external server test login to Gmail. Simple as that.

They manually update that status page to not scare away stockholders.

Out of curiosity, what response time do you expect on a page like that? And what level of detail? I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.

> what response time do you expect on a page like that?

Faster than a free third-party website’s response time. Google should know they are down and tell people about it before Hacker News, Twitter, etc. Google should be the source of truth for Google service status, not social media.

> And what level of detail?

Enough to not tell people that there are “No issues” with services.

> I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.

Google employs enough people to do both.

I'd expect that status page to be automated based on a number of metrics / health checks. Our equivalent is.

I'd expect within seconds that Google is alerted of a very large number of issues with their servers and that the status page would be updated (the green light going to red) within seconds. It's now quite some time after the start of the outage and everything is still green on that status page.

How many employees do you think Google has? Do you think they're all working on the same task?

Well, probably a lot of them are sitting around doing nothing because Google's down right now.

It's a big group of teams... one team is responsible for monitoring (And status reporting) alone

> I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.

It's not like they would be working on the status page right now, that work should have been done a long time ago...

If they know its broken, one of the _many_ engineers/support across Youtube+Gmail+etc that are all known to be down related to this bug should be able to update it in first few minutes. Especially if this isn't a 5 min fix.

It should be faster than it takes TechCrunch to write an article about it

TechCrunch don't need to verify anything.

There should be enough people on that to communicate out as well as resolve.