News for hackers. Stuff that matters.
Re:Google Cloud Networking reporting issues by mavelikara on Wednesday March 17, 2021 @01:36PM (#26494407)
Re:Google Cloud Networking reporting issues by alexdumitru on Wednesday March 17, 2021 @01:43PM (#26494514)
Everything Google-related is down for me, so I can't open that page.
Re:Google Cloud Networking reporting issues by pentium10 on Wednesday March 17, 2021 @01:58PM (#26494716)
The issue started occurring intermittently at 08:26 US/Pacific. The issue is impacting Google’s Backbone network and may impact various services when accessing them from a different region or from the internet. Impacted services include Cloud Services (Workspace, Firebase, GCP) as well as other Google properties.

Connectivity within a zone should not be impacted.

Our engineering team has implemented a mitigation and is now monitoring the effectiveness of the change.

Re:Google Cloud Networking reporting issues by paxys on Wednesday March 17, 2021 @02:24PM (#26495009)
Hosting your status page on the same infrastructure it is reporting on is the most idiotic thing a service can do.
Re:Google Cloud Networking reporting issues by tpmx on Wednesday March 17, 2021 @02:31PM (#26495086)
Seems like the status page was on separate infra (I could access it while the GCP services were down), but Google Public DNS (8.8.8.8) was also down.

Perhaps alexdumitru was using 8.8.8.8?

Re:Google Cloud Networking reporting issues by alexdumitru on Thursday March 18, 2021 @10:01PM (#26509538)
You're right, I'm using 8.8.8.8.
Re:Google Cloud Networking reporting issues by macintux on Wednesday March 17, 2021 @02:25PM (#26495019)
It's tough, though. The PR embarrassment of hosting your Google Cloud status page on AWS, say, would be substantial.
Re:Google Cloud Networking reporting issues by dilyevsky on Wednesday March 17, 2021 @02:31PM (#26495083)
I’m sure google can afford some colo space (probably would be cheaper too)
Re:Google Cloud Networking reporting issues by dylan604 on Wednesday March 17, 2021 @02:35PM (#26495133)
Makes me think of a calculus word problem: As the things Google cannot afford approaches 0...
Re:Google Cloud Networking reporting issues by macintux on Wednesday March 17, 2021 @02:35PM (#26495131)
Sure, but what’s more likely to be reliable: GCP/AWS or a colo?

It’s also embarrassing (and can cause stress for your customers) when your status page is down.

Re:Google Cloud Networking reporting issues by dilyevsky on Wednesday March 17, 2021 @02:49PM (#26495307)
Imo for simple deployment colo with a major provider (equinix, coresite, etc) and redundant transit beats any cloud on reliability hands down
Re:Google Cloud Networking reporting issues by jeffbee on Wednesday March 17, 2021 @03:20PM (#26495698)
How do you know that there isn't a disaster plan under which Google routes requests for status.cloud.google.com to some other, non-Google host?
Re:Google Cloud Networking reporting issues by dilyevsky on Wednesday March 17, 2021 @03:58PM (#26496094)
I honestly have no idea, hope they do. My point is aws is not the only alternative (and not even the best one) for setting this up.
Re:Google Cloud Networking reporting issues by closeparen on Wednesday March 17, 2021 @02:43PM (#26495232)
A separate failure domain / uncorrelated failure can be more important than the absolute rate of failure.
Re:Google Cloud Networking reporting issues by CSDude on Wednesday March 17, 2021 @02:13PM (#26494888)
Why the most GCP outages are global? Even though it says issue started in US/Pacific, I'm not sure if it affects other regions and the status page of them does not make it clear.
Re:Google Cloud Networking reporting issues by advisedwang on Wednesday March 17, 2021 @02:42PM (#26495226)
"US/Pacific" is detailing the timezone they are using for start/end times. It's not saying the issue is only in that location.
Re:Google Cloud Networking reporting issues by merb on Wednesday March 17, 2021 @02:17PM (#26494927)
they aren't, everything from germany works.
Re:Google Cloud Networking reporting issues by notyourday on Wednesday March 17, 2021 @02:29PM (#26495055)
Because Google engineers think that they are the smartest people in the world which means they don't actually engineer for minimization of a blast radius.

You should just see how Google builds the prefix lists that it allows customers to advertise to Google via PNIs. You would think they are building it off some registry lists because why else would Google insists that the routes are registered? Oh no, that's for small people. At Google they just bring up a session to you, let you advertise and take whatever you advertised to them during the setup as the allow list because everything is done via custom automated software!

Re:Google Cloud Networking reporting issues by opsunit on Wednesday March 17, 2021 @02:53PM (#26495357)
If you are unsure of the tenancy of any given site https://runson.cloud may be of some help.
Re:Google Cloud Networking reporting issues by naranha on Wednesday March 17, 2021 @02:41PM (#26495214)
> Workaround: None at this time.

I wish I could say that to my customers...

Re:Google Cloud Networking reporting issues by 1023bytes on Wednesday March 17, 2021 @01:43PM (#26494522)
My home internet connection is down, maybe some larger scale backbone problem?
Re:Google Cloud Networking reporting issues by NicoJuicy on Wednesday March 17, 2021 @01:48PM (#26494580)
Dns on 8.8.8.8 perhaps?
Re:Google Cloud Networking reporting issues by 1023bytes on Wednesday March 17, 2021 @01:49PM (#26494592)
Oh that's it, thanks
Re:Google Cloud Networking reporting issues by freedomben on Wednesday March 17, 2021 @01:58PM (#26494721)
Does Cloudflare use Google Cloud? I know they have a ton of their own hardware, but maybe they outsource some things?

I had my first ever Cloudflare outage earlier today. It's just a free site that has a single page rule that redirects to a different domain, but it started refusing connections for a little while today.

Re:Google Cloud Networking reporting issues by pubg on Wednesday March 17, 2021 @02:06PM (#26494806)
No.
Re:Google Cloud Networking reporting issues by Thaxll on Wednesday March 17, 2021 @02:33PM (#26495112)
That's why WoW is down probably.
Re:Google Cloud Networking reporting issues by packetslave on Wednesday March 17, 2021 @02:35PM (#26495123)
WoW is self-hosted, or at least they were a few years ago. Racks and racks of HP blade servers.
Re:Google Cloud Networking reporting issues by ihsw on Wednesday March 17, 2021 @02:40PM (#26495194)
Re:Google Cloud Networking reporting issues by packetslave on Wednesday March 17, 2021 @03:06PM (#26495514)
Nothing in that press release mentions anything about WoW
Re:Google Cloud Networking reporting issues by mywacaday on Wednesday March 17, 2021 @02:35PM (#26495122)
Also LinkedIn android app
Re:Google Cloud Networking reporting issues by alexdumitru on Wednesday March 17, 2021 @01:42PM (#26494504)
All Google services have been down for me for the past 30 minutes, at least.
Re:Google Cloud Networking reporting issues by gcbirzan on Wednesday March 17, 2021 @01:55PM (#26494667)
Try not using 8.8.8.8, it's down from some places.
Re:Google Cloud Networking reporting issues by toomuchtodo on Wednesday March 17, 2021 @01:48PM (#26494582)
Miami: Google services (Mail, Drive, Calendar, Maps, DNS) available to me.
Re:Google Cloud Networking reporting issues by tpmx on Wednesday March 17, 2021 @02:11PM (#26494854)
Sweden: Google's public web-based services all seem to be available, GCP services are not. Except the console.

Edit: things started working now.

Re:Google Cloud Networking reporting issues by booleanbetrayal on Wednesday March 17, 2021 @01:50PM (#26494599)
Seeing API calls to Sentry fail - https://status.sentry.io/
Re:Google Cloud Networking reporting issues by smithcoin on Wednesday March 17, 2021 @01:34PM (#26494384)
Brutal. We are seeing requests that originate from heroku time out because response times are well over 30 seconds.
Re:Google Cloud Networking reporting issues by endlessvoid94 on Wednesday March 17, 2021 @01:41PM (#26494495)
Same. It took over an hour for this to show up as an incident somewhere. I was going crazy.
Re:Google Cloud Networking reporting issues by gundmc on Wednesday March 17, 2021 @02:38PM (#26495172)
Seems to be resolved according to the same source.
Re:Google Cloud Networking reporting issues by dboreham on Wednesday March 17, 2021 @02:35PM (#26495128)
s/reporting/is reporting/ ?
Re:Google Cloud Networking reporting issues by vntok on Wednesday March 17, 2021 @01:51PM (#26494608)
OVH last Wednesday, Oracle the very next day, Azure yesterday, GCP today... will a crime syndicate employing arsonists as well as hackers claim all these in a month as a demonstration of their capabilities?
Re:Google Cloud Networking reporting issues by throwaway53453 on Wednesday March 17, 2021 @02:11PM (#26494859)
These comments crop up regularly on outage posts. There's probably some psychological bias that explains it but I can't remember... I think the last time Google went down there were comments blaming it on Summer Interns :)
Re:Google Cloud Networking reporting issues by dmlittle on Wednesday March 17, 2021 @02:30PM (#26495069)
No, it was SolarWinds that blamed the intern for their security issues.

In all seriousness... if a single employee can accidentally single-handedly screw things up you've got a process problem. There should be checks and balances to make sure that what you're trying to do make sense. These might be in the form of a code-review, automated validation, a staging environments, etc. Yes, these might be things that don't exist at a scrappy startup and that is fine as they'll get there eventually but for any large company not having good processes is reckless.

Re:Google Cloud Networking reporting issues by throwaway53453 on Wednesday March 17, 2021 @03:23PM (#26495734)
These issues happen despite all those things. You can bet that Google has that and more in place. Just like in aerospace engineering, things go wrong when dozens of failures happen across the critical path, not any individual piece.
Re:Google Cloud Networking reporting issues by dmlittle on Wednesday March 17, 2021 @03:30PM (#26495806)
Yes, and issues will always occur. What I was trying to point out that it's never an individual's fault alone. Sure someone might have tripped the final domino but they might not have had anything to do with everything else that happened. That's why blameless postmortems are a thing. Blameless (to me) doesn't mean that you don't name any person but rather that there's no blame assigned to anyone. I want to know what Person A did to cause the incident but there shouldn't be any retribution against Person A.
Re:Google Cloud Networking reporting issues by rzzzt on Wednesday March 17, 2021 @02:58PM (#26495415)
Frequency illusion aka. Baader-Meinhof phenomenon? https://en.wikipedia.org/wiki/Frequency_illusion
Re:Google Cloud Networking reporting issues by dylan604 on Wednesday March 17, 2021 @02:38PM (#26495169)
Anybody know the current where abouts of Mr. Robot?