News for hackers. Stuff that matters.
Re:Google Outage in Europe by kroolik on Friday November 12, 2021 @05:39AM (#29197963)
There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made.

https://en.m.wikipedia.org/wiki/Google_services_outages

Re:Google Outage in Europe by delroth on Friday November 12, 2021 @06:12AM (#29198121)
Perf has been over for almost a month now, and the evaluation period was over more than two months ago.
Re:Google Outage in Europe by dspillett on Friday November 12, 2021 @07:43AM (#29198743)
> and the evaluation period was over more than two months ago

Excellent time to slack off a bit, make a few mistakes, then come next eval you can point to a marked improvement over the intervening six-to-nine months!

Re:Google Outage in Europe by roenxi on Friday November 12, 2021 @08:44AM (#29199161)
The line of logic this thread has followed so far suggests there will be a reason for a Google outage every month of the year.
Re:Google Outage in Europe by NikxDa on Friday November 12, 2021 @09:53AM (#29199796)
Couldn‘t have phrased it any better.
Re:Google Outage in Europe by shuckles on Friday November 12, 2021 @11:51AM (#29201444)
I believe the term of art is “just so story.”
Re:Google Outage in Europe by kroolik on Friday November 12, 2021 @06:13AM (#29198131)
Then it might, indeed, be unrelated. Thanks for clarification!
Re:Google Outage in Europe by samhw on Friday November 12, 2021 @06:55AM (#29198405)
Could it not be knock-on effects of earlier changes? Or indeed post-review changes from scared/precarious engineers?
Re:Google Outage in Europe by mucle6 on Friday November 12, 2021 @07:00AM (#29198458)
I think we are exactly mid way between the end year 2021 perf and mid year 2022 perf, so its not clear to me that this could be to blame
Re:Google Outage in Europe by skinkestek on Friday November 12, 2021 @07:24AM (#29198631)
That explains:

The people who did the stuff have either gotten their promotions and moved on or didn't get a promotion and have given up.

:-)

Re:Google Outage in Europe by oblio on Friday November 12, 2021 @05:54AM (#29198036)
He he. It's kind of hilarious. Maybe they should stagger reviews to ensure high availability? :-D
Re:Google Outage in Europe by kroolik on Friday November 12, 2021 @05:58AM (#29198057)
Nothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced.

My post is just a speculation, let's wait for the actual technical details doc from Google.

Re:Google Outage in Europe by oblio on Friday November 12, 2021 @06:24AM (#29198194)
Well, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made.

Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.

Re:Google Outage in Europe by yunohn on Friday November 12, 2021 @06:52AM (#29198382)
> logical conclusion regarding the spiky yearly activity

Why is it logical? There’s tons of changes being deployed at all times, at all large companies.

Across products, verticals, everything - hundreds of changes at any given point. Some of these changes can introduce hard-to-predict bugs in globally distributed systems. Most of the time, external users don’t even notice before they’re fixed.

Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies.

Re:Google Outage in Europe by michaelt on Friday November 12, 2021 @08:25AM (#29199031)
Hypothetically speaking? 11 months a year there's no incentive to cut corners and only do 2 weeks of testing on something that really needs 3. If one month a year rushing things out with a bit less testing is rewarded, I can believe some people would respond to that.

Of course, I wouldn't go as far as to call this a "logical conclusion" as the evidence I've seen is very slim.

Re:Google Outage in Europe by fragmede on Friday November 12, 2021 @02:07PM (#29203284)
Except that there are two a year at Google., so it's 10 months of the year, 5 and 5, and Covid's made everything weird.
Re:Google Outage in Europe by oblio on Friday November 12, 2021 @07:21AM (#29198609)
> Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies.

When are they? My bet is that they're roughly around the same time period, even if they don't start or close on December 31st.

Re:Google Outage in Europe by UncleMeat on Friday November 12, 2021 @07:24AM (#29198628)
It can’t be logical because it is based on bad facts.

We wrote evals in August. Anybody racing to get things launched before perf did so months ago. The timing described in the top post is just false.

We write the next one in February. We are almost exactly half way between perf cycles. Your hunch that this lines up with the end of the year is false.

Re:Google Outage in Europe by kixiQu on Friday November 12, 2021 @11:44AM (#29201315)
Far more likely IMO to coincide with trying to get Exciting New Features out before release freezes for some conference or other.
Re:Google Outage in Europe by bamboozled on Friday November 12, 2021 @05:43PM (#29205383)
There is also a lot of pressure on engineers around black Friday and Cyber Monday to get things done before any change embargoes come into place. This is coming from someone who just worked 16 hours straight to scale things before a change freeze.

I don't know how or if this would impact Google, but I'm sure someone there has at least thought about those dates.

Re:Google Outage in Europe by trhway on Friday November 12, 2021 @07:22AM (#29198618)
November 11 is a public holiday in US. And it seems it has in recent years became a company holiday in many companies too. May be that break in established over many years cadence of workday and holidays plays a role here, like say affecting smoothness of support transition between US-EMEA-APJ.
Re:Google Outage in Europe by mikelward on Friday November 12, 2021 @09:33AM (#29199595)
Veterans Day is not a day off for US Googlers.
Re:Google Outage in Europe by alpb on Tuesday November 16, 2021 @03:55PM (#29246247)
> evaluation and promotion window ends around december

Googler here, that is not when those promotion/perf evaluation windows happen.

Re:Google Outage in Europe by inglor on Friday November 12, 2021 @06:49AM (#29198352)
I would be very interested in any research about code quality with relation to promotion packets. Ideally acamdemic.

I am not sure where to look for it

Re:Google Outage in Europe by rajin444 on Friday November 12, 2021 @11:31AM (#29201094)
Is there an objective way to measure code quality?

We barely understand the systems we’re writing code for - I don’t see how you could objectively judge the code to manage those systems without fully understanding the systems first.

You could have wave some metrics like number of bugs or test coverage, but I can’t think of any (aforementioned included) that aren’t subject to tons of confounding variables.

Re:Google Outage in Europe by de6u99er on Friday November 12, 2021 @06:59AM (#29198448)
Some people are definitely not gatting a raise.
Re:Google Outage in Europe by jankeymeulen on Friday November 12, 2021 @07:12AM (#29198536)
More likely someone actually will. A blameless postmortem will be written, and the people that will fix the bug or systems issue will have something to work on with high visibility and high impact, which tends to translate in good performance ratings.

(Googler, opinions are my own)

Re:Google Outage in Europe by radishingr on Saturday November 13, 2021 @01:22AM (#29207868)
"Impact: fixed processes that led to 8 hr outage" seems like an easy case to make.
Re:Google Outage in Europe by deanCommie on Friday November 12, 2021 @12:47PM (#29202289)
And this is why nobody serious should build on GCP.

Google's more interested in placating their primadonna engineers than solving customer problems.

Re:Google Outage in Europe by moffkalast on Friday November 12, 2021 @05:59AM (#29198066)
Then they get fired instead, well done ha.
Re:Google Outage in Europe by evercast on Friday November 12, 2021 @06:12AM (#29198127)
Comments like this make me wonder if people really expect engineers to be fired because of an outage? I do not work at Google, but none of my workplaces would fire engineers because of a failure. Mistakes happen. As long as they are not repeated, everything is good.

If your company fires people in situations like this, run away and never look back.

Re:Google Outage in Europe by ddalex on Friday November 12, 2021 @06:18AM (#29198153)
Googler here, not speaking on the behalf of the company, my opinions are my own

People do absolutely NOT get fired over incidents. Making mistakes is human. An incident will prompt a review of the systems and safeguards in place to prevent such an incident, much like an airline incident investigation -

basically "somebody fat-fingered it" is never the answer, postmortems are always blameless

EDIT: now that I think of it, the opposite thing happens after a major incident - a systemic failure should be identified, people are being hired to fix it :)

Re:Google Outage in Europe by xcambar on Friday November 12, 2021 @06:44AM (#29198310)
> Googler here, not speaking on the behalf of the company, my opinions are my own

Why do employees at big tech names (FAANG et al.) are so often so cautious as to include this as a foreword everywhere? Twitter bios are full of that, for instance.

It is crazy to me; who would expect anything else that our opinions being your own and nothing more? Who would expect that your word (with all due respect) is worth anything with regards to the company's PR?

Is there an actual risk in the US? Have there been trials or anything that push people to add such statements?

Re:Google Outage in Europe by ddalex on Friday November 12, 2021 @07:01AM (#29198465)
Can't speak for other companies, but this is covered in basic training at Google - if you're not authorized to speak on behalf of the company, you must make it clear when your writings may be mistaken or constructed to represent the company.

Basically the company has specially trained people that speak on behalf of the company, and that message should not be confounded by personal opinions of other employees. For example, on the recent FB outage, there was an employee posting inside information on reddit - media companies just took it at face value and ran around with it reporting as it was what FB itself was saying about the outage.

I'm not aware of any actual risks in the US, but then again I'm not in the US. For me this seems a minor point, and I actually enjoy separating my public persona from the company for which I work, being it Google or a small startup.

Re:Google Outage in Europe by jefftk on Friday November 12, 2021 @10:21AM (#29200079)
> On the recent FB outage, there was an employee posting inside information on reddit - media companies just took it at face value and ran around with it reporting as it was what FB itself was saying about the outage.

To be fair, I think the media would have done that even with a "speaking only for myself" disclaimer.

Re:Google Outage in Europe by secondaryacct on Friday November 12, 2021 @01:50PM (#29203083)
Every company says that, the obvious solution is to never name it when you speak. Why do these people need to say "Im a googler" and then immediately "but forget it, I speak on my own"... obviously there s value in the fact they re at Google and it will color their discourse which is already probably forbidden.

Dont name your company if you intend to speak for yourself.

Re:Google Outage in Europe by DnDGrognard on Friday November 12, 2021 @12:10PM (#29201752)
It's the same at all large companies - its CYA boiler plate.

Though I almost got to be the official spokesperson for British Telecom responding on the alt.2600 news group about the the Met police VMB hack - press office was cool but the internal security was not.

Re:Google Outage in Europe by xcambar on Friday November 12, 2021 @10:24AM (#29200110)
Would you say: "I live in [city], opinions are my own", or "I am married to [person], opinions are my own"?

If no, why are you doing this for the company you're trading skills and time against money?

Re:Google Outage in Europe by LadyCailin on Friday November 12, 2021 @06:57AM (#29198427)
I work at a large tech company, and they do mention in the on boarding materials that we represent the company, so we should be careful in our social media profiles. My solution to this is to not associate my social media profiles with my employer. This is technically not really what we’re supposed to do, and I might have to change that approach at some point if I move high enough in the org to start getting attention from people, but this works for me better than disclaimers on all my posts.
Re:Google Outage in Europe by riffic on Friday November 12, 2021 @01:07PM (#29202541)
if they wanted to control this so bad they'd provision you a managed account like how email addresses are managed.
Re:Google Outage in Europe by secondaryacct on Friday November 12, 2021 @01:56PM (#29203152)
Yet all these googlers breached it immediately in the first sentence by naming their company...
Re:Google Outage in Europe by xcambar on Friday November 12, 2021 @02:57PM (#29203854)
Sarcastically, "I'm a googler, opinions my own" reads a lot like "I'm a googler, just so you know".

I didn't want to emphasize on that on the first comment yet to be honest, I find it pedantic because it's pointless, legally speaking.

Re:Google Outage in Europe by null_object on Friday November 12, 2021 @07:02AM (#29198479)
> Why do employees at big tech names (FAANG et al.) are so often so cautious as to include this as a foreword everywhere?

This isn’t just ‘big tech’ - I work at a relatively small tech company, but I’d never want anything I say about the company to be mistaken as some sort of ‘official statement’ especially if it related to an incident that possibly had a financial impact on external parties, and could conceivably be misused in that context in the future.

I go as far as never writing private emails from my work mail for the same sort of reasons - although that is from a possibly over-abundance of caution.

Re:Google Outage in Europe by b0afc375b5 on Friday November 12, 2021 @06:54AM (#29198396)
It's in the spirit of full disclosure, which some, including me, appreciate.
Re:Google Outage in Europe by Rebelgecko on Friday November 12, 2021 @12:39PM (#29202186)
Part of it is because the company asks us to. Part of it is because I think it's reasonable to tell people your biases, and it can avoid the situation where substantive conversation gets derailed by "gotchas". If I make a comment about how I think Google Meet has the best noise cancellation of any video chat software, even though I don't work on Meet or anything adjacent to it, it's still a bad look if someone can dig through my comment history and pull out a previous comment about how I work for Google.
Re:Google Outage in Europe by com2kid on Tuesday November 16, 2021 @03:20PM (#29245887)
> It is crazy to me; who would expect anything else that our opinions being your own and nothing more?

Rumor mill journalists will mine social media and forum comments and write entire articles about "so and so FAANG employee gives hint at future merger" when some dev comments how much they've enjoyed using some library recently.

Re:Google Outage in Europe by fshbbdssbbgdd on Friday November 12, 2021 @03:35PM (#29204225)
They want to mention their employer to gain authority in the discussion, but since mentioning their employer is a legal/PR risk, they need to follow it up with a disclaimer (this only partially mitigates the risk, but it’s worth it to get the brag in).
Re:Google Outage in Europe by secondaryacct on Friday November 12, 2021 @01:46PM (#29203030)
It s because they only hire idiots at Google. Im from a big company, I just dont name it and assume humans understand my opinions are my own :D
Re:Google Outage in Europe by gsich on Friday November 12, 2021 @08:21AM (#29198996)
Especially when it's not an opinion.
Re:Google Outage in Europe by samhw on Friday November 12, 2021 @09:29AM (#29199559)
In fairness, 'opinion' is a horrendously vague and ill-defined word. It does double duty as (i) 'normative value' and (ii) 'personal understanding of the descriptive facts', which two senses are constantly confused - for example right here.

That's why we constantly get "it's just my opinion" used in reference to type-ii opinions (personal understanding of descriptive fact), when it's only really appropriate to type-i opinions (normative value).

Many conversations would be far clearer if it were abandoned in favour of more precise language, IMPUOTDF.

Re:Google Outage in Europe by not1ofU on Friday November 12, 2021 @04:46PM (#29204908)
"Many conversations would be far clearer if it were abandoned in favour of more precise language, IMPUOTDF"... um whats IMPUOTDF? I did try to google it, but only this post was found.
Re:Google Outage in Europe by samhw on Sunday November 14, 2021 @11:36AM (#29218246)
Sorry, I was joking: 'in my personal understanding of the descriptive facts', referring back to that second definition of 'opinion' earlier in the comment.
Re:Google Outage in Europe by samhw on Friday November 12, 2021 @07:40AM (#29198728)
Yeah, this 'blameless' ethos has definitely trickled down from FAANG to decently-sized decently-reputed places I've worked at - and certainly to #EngTwitter.

I think it's a bit over-applied in some cases. Does it not commit you to the theorem that every process can be made so perfect as to be completely invulnerable to one human being making a mistake? (At least, in the form exemplified by the common tweets to the effect that "your processes are to blame for $incident, not your interns/engineers/etc".)

Even if you required two-person auth for every single thing, two people will make a mistake now and then, and in reality - due to our being social animals - the two probabilities are not truly independent.

I just don't see how this is feasible in reality. A more realistic principle feels like: "people will infrequently make mistakes, and that's of course natural and human and forgiveable, but far fewer incidents should be vulnerable to human error than currently are".

Re:Google Outage in Europe by assbuttbuttass on Friday November 12, 2021 @09:26AM (#29199532)
I of course agree that mistakes are inevitable. That being said, the point of blameless culture is not to make a process invulnerable to mistakes. Instead during a post-mortem, we look at how to prevent that particular incident from happening again.
Re:Google Outage in Europe by fragmede on Friday November 12, 2021 @02:10PM (#29203336)
You're totally right and the SRE book by Google goes over this - the company's culture does not allow firing people for outages. If you're somewhere this still happens, run away (or you'd better be getting paid more than top-end ICs at Google.
Re:Google Outage in Europe by kroolik on Friday November 12, 2021 @06:02AM (#29198077)
Why would you fire an engineer you have just spent millions to train?
Re:Google Outage in Europe by chrisseaton on Friday November 12, 2021 @06:38AM (#29198275)
What tech company spends millions on training anyone?
Re:Google Outage in Europe by pestaa on Friday November 12, 2021 @06:42AM (#29198298)
It's reframing lost revenue, not talking about literal training cost.
Re:Google Outage in Europe by 1ibsq on Friday November 12, 2021 @06:48AM (#29198349)
The other comments already explained it, but I'm wondering how you haven't come across this 'saying' before. It's so overused and also cheesy in my opinion.
Re:Google Outage in Europe by stackbutterflow on Friday November 12, 2021 @12:00PM (#29201601)
People are born every day. Every day tens of thousands of people will hear about hacker news, the pyramids, Darth Vader being someone's father, for the first time.
Re:Google Outage in Europe by 1ibsq on Monday November 15, 2021 @05:55AM (#29225952)
It was not meant ill-willed. I was just wondering. Regarding what you said, I think I understand what your point is. On the other hand it makes some difference among whose people you ask - some things are much likelier to be common sense (or at least heard of) than in other places. Whatever.
Re:Google Outage in Europe by chrisseaton on Friday November 12, 2021 @07:01AM (#29198464)
Lol maybe if people actually did spend millions training their people up front we could do better?
Re:Google Outage in Europe by exikyut on Friday November 12, 2021 @08:34AM (#29199089)
139,995 employees at Google * 1,000,000 = $139,995,000,000

$140 billion dollars. On training.

On the one hand... you know what, I'd love to work in an environment like that. Seriously.

On the other hand... what's the argument you make to the CFO in support of this? Honest question, interested to hear answers.

Re:Google Outage in Europe by chrisseaton on Friday November 12, 2021 @08:49AM (#29199203)
I work part time in the Army. In the Army when you go from their equivalent of junior to mid-level they take you out of your job for eight months of dedicated personal development, before you start your first mid-level job. When you go to their equivalent of senior they take you out for a year.

I don't know how much that costs all-in, including the salaries, instructors, facilities, but might be starting to approach a million.

That's valuing training!

Re:Google Outage in Europe by secondaryacct on Friday November 12, 2021 @02:01PM (#29203214)
But the army is a cost center! The workers have some money shaved off their salary to pay for an army that allows delinquents and half-disabled to pew-pew guns in the forest, leaving them in peace. It's not comparable to a productive entreprise that needs to build things for people or perish.

For instance, if Google fails and cant profit, it cant just shoot at their client until they pay. Your organisation can.

Re:Google Outage in Europe by chrisseaton on Friday November 12, 2021 @02:16PM (#29203410)
> It's not comparable to a productive entreprise that needs to build things for people or perish.

Well we had to build an Army to win against fascism in the Second World War or we all would have perished.

And 'perished' means literally dead or subject to fascism, not just going out of business.

Re:Google Outage in Europe by confidantlake on Friday November 12, 2021 @03:24PM (#29204114)
WW2 was 70+ years ago. The USSR fell 30+ years ago. Military budgets are still incredibly high. The army as it is today does not need to be that efficient. And even during WW2 times the US did not face a credible threat of invasion. The last time the US faced an invasion on the mainland was during the war of 1812.
Re:Google Outage in Europe by chrisseaton on Friday November 12, 2021 @03:58PM (#29204447)
> The army as it is today does not need to be that efficient.

You want the Army to be... less efficient? Spend more for less capability?

Re:Google Outage in Europe by confidantlake on Friday November 12, 2021 @04:29PM (#29204759)
No I mean that for the US army today the downside to the army being inefficient is that money is wasted. Not great, but not a disaster. For a different country that could mean the country gets invaded or the government collapses (like Afghanistan).
Re:Google Outage in Europe by handrous on Friday November 12, 2021 @12:45PM (#29202264)
Middle and upper management get there via connections and picking things up on their own. It's unsurprising that they don't want to be subjected to competition with lesser people who can "merely" be trained to do their job as well, or better.
Re:Google Outage in Europe by dolmen on Friday November 12, 2021 @09:01AM (#29199314)
Some jobs can be learned on the job.

But I'm glad that learning to kill people (military) is not taught that way.

Re:Google Outage in Europe by chrisseaton on Friday November 12, 2021 @09:12AM (#29199396)
If your job is something like a staff officer in a Brigade, you could learn that on the job, the Google way, because exercising is also 'the job', but they don't get you to do that - instead they take the time to fully pull you out of all work commitments for dedicated personal development. These periods of personal development are about personal skills rather than combat training, which you've already done by this point.
Re:Google Outage in Europe by dukeyukey on Friday November 12, 2021 @06:36PM (#29205782)
Someone who just cost their company millions in revenues is gonna be _extremely_ careful not to make the same mistake again. Hence, million-dollsr training.
Re:Google Outage in Europe by pfarrell on Friday November 12, 2021 @06:49AM (#29198351)
The GP is a reference to the anecdote about IBM’s Thomas Watson not firing a executive who had made an error costing the company a substantial amount of money.
Re:Google Outage in Europe by l33t2328 on Friday November 12, 2021 @06:42AM (#29198293)
The implication is that millions were spent training the person who made the mistake when they cost the company millions by making that mistake.
Re:Google Outage in Europe by Ueland on Friday November 12, 2021 @06:42AM (#29198296)
An outage can be pretty expensive but it is training for those whom triggered it and/or those that fixes it.
Re:Google Outage in Europe by goldcd on Friday November 12, 2021 @06:04AM (#29198083)
I'd guess because Europe-wide outages are costing more than millions
Re:Google Outage in Europe by cranekam on Friday November 12, 2021 @06:33AM (#29198255)
But firing someone doesn't undo an incident. It just introduces other weird incentives. People become afraid to change things for fear of breaking something, or when something does break they try to hide it rather than feeling like they can immediately ask for and receive help to fix it.

The only time someone should be fired for causing an outage is if they're negligent or sloppy or mess things up all the time. This is rare. Almost always outages in large systems are the combination of many factors — latent bugs, design flaws, abnormal load, etc, any one or two of which wouldn't take the site down. But when the combine in a perfect storm that nobody foresaw things fall over.

Re:Google Outage in Europe by kreeben on Friday November 12, 2021 @06:10AM (#29198111)
But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success.

100% of all devs make huge mistakes, at least once.

Re:Google Outage in Europe by KronisLV on Friday November 12, 2021 @06:15AM (#29198140)
> But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes.

I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues.

Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite.

In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.

Re:Google Outage in Europe by kreeben on Friday November 12, 2021 @06:24AM (#29198189)
N+1 issues aren't nearly as devastating as N^2. Commend them for not putting your systems to a complete halt, then teach them how to reason about this properly.

>> a quality gate

Yes, this, also.

Re:Google Outage in Europe by KronisLV on Friday November 12, 2021 @06:50AM (#29198371)
> N+1 issues aren't nearly as devastating as N^2.

Depends on the project, i guess: if you're unlucky enough to be working on a monolith and suddenly a page takes 5'000 SQL queries to load as opposed to 100, because someone thought that initializing data through service/DB calls in a loop is "easier" than writing views in the DB, it might still kill the entire system anyways, depending on the count of users.

And once this data initialization is sufficiently complicated and convoluted for you not to be able to rewrite it and them not wanting to rewrite it, all while "the business" is breathing down on your neck, you might either want to introduce caching (and possibly run into cache invalidation problems down the road), or just freshen up your CV.

I guess i'd also like to expand on the previous suggestion and advise others to consider performance/load testing as well, especially when coupled with APM solutions like Skywalking or even Matomo analytics, both of which can allow you to aggregate the historical page load times, CPM and overall performance of your applications, to figure out what went wrong when.

Re:Google Outage in Europe by kroolik on Friday November 12, 2021 @06:10AM (#29198112)
Still, that engineer (if its engineer's fault) is extremely unlikely to make that issue again. IMHO, the problem is systemic in that why the system allows such errors (if its human error) to happen. Given Google's scale I think a lot of the generally known scenarios are covered and what you see is tens of services interacting in not obvious ways. Those unobvious scenarios manifest in situations like this.
Re:Google Outage in Europe by hdjjhhvvhga on Friday November 12, 2021 @06:45AM (#29198321)
It makes no sense at all. After the outage you have not only a review of the causes and appropriate remedies, but also more experienced people who are now more aware of possible consequences of seemingly unrelated actions and will take extra care not to make these things happen in the future.

Also, such cases are rarely the "fault" of a single person. Or, the direct/immediate cause is often not the main one.