What tech company spends millions on training anyone?

It's reframing lost revenue, not talking about literal training cost.

The other comments already explained it, but I'm wondering how you haven't come across this 'saying' before. It's so overused and also cheesy in my opinion.

People are born every day. Every day tens of thousands of people will hear about hacker news, the pyramids, Darth Vader being someone's father, for the first time.

It was not meant ill-willed. I was just wondering. Regarding what you said, I think I understand what your point is. On the other hand it makes some difference among whose people you ask - some things are much likelier to be common sense (or at least heard of) than in other places. Whatever.

Lol maybe if people actually did spend millions training their people up front we could do better?

139,995 employees at Google * 1,000,000 = $139,995,000,000

$140 billion dollars. On training.

On the one hand... you know what, I'd love to work in an environment like that. Seriously.

On the other hand... what's the argument you make to the CFO in support of this? Honest question, interested to hear answers.

I work part time in the Army. In the Army when you go from their equivalent of junior to mid-level they take you out of your job for eight months of dedicated personal development, before you start your first mid-level job. When you go to their equivalent of senior they take you out for a year.

I don't know how much that costs all-in, including the salaries, instructors, facilities, but might be starting to approach a million.

That's valuing training!

But the army is a cost center! The workers have some money shaved off their salary to pay for an army that allows delinquents and half-disabled to pew-pew guns in the forest, leaving them in peace. It's not comparable to a productive entreprise that needs to build things for people or perish.

For instance, if Google fails and cant profit, it cant just shoot at their client until they pay. Your organisation can.

> It's not comparable to a productive entreprise that needs to build things for people or perish.

Well we had to build an Army to win against fascism in the Second World War or we all would have perished.

And 'perished' means literally dead or subject to fascism, not just going out of business.

WW2 was 70+ years ago. The USSR fell 30+ years ago. Military budgets are still incredibly high. The army as it is today does not need to be that efficient. And even during WW2 times the US did not face a credible threat of invasion. The last time the US faced an invasion on the mainland was during the war of 1812.

> The army as it is today does not need to be that efficient.

You want the Army to be... less efficient? Spend more for less capability?

No I mean that for the US army today the downside to the army being inefficient is that money is wasted. Not great, but not a disaster. For a different country that could mean the country gets invaded or the government collapses (like Afghanistan).

Middle and upper management get there via connections and picking things up on their own. It's unsurprising that they don't want to be subjected to competition with lesser people who can "merely" be trained to do their job as well, or better.

Some jobs can be learned on the job.

But I'm glad that learning to kill people (military) is not taught that way.

If your job is something like a staff officer in a Brigade, you could learn that on the job, the Google way, because exercising is also 'the job', but they don't get you to do that - instead they take the time to fully pull you out of all work commitments for dedicated personal development. These periods of personal development are about personal skills rather than combat training, which you've already done by this point.

Someone who just cost their company millions in revenues is gonna be _extremely_ careful not to make the same mistake again. Hence, million-dollsr training.

The GP is a reference to the anecdote about IBM’s Thomas Watson not firing a executive who had made an error costing the company a substantial amount of money.

The implication is that millions were spent training the person who made the mistake when they cost the company millions by making that mistake.

An outage can be pretty expensive but it is training for those whom triggered it and/or those that fixes it.

I'd guess because Europe-wide outages are costing more than millions

But firing someone doesn't undo an incident. It just introduces other weird incentives. People become afraid to change things for fear of breaking something, or when something does break they try to hide it rather than feeling like they can immediately ask for and receive help to fix it.

The only time someone should be fired for causing an outage is if they're negligent or sloppy or mess things up all the time. This is rare. Almost always outages in large systems are the combination of many factors — latent bugs, design flaws, abnormal load, etc, any one or two of which wouldn't take the site down. But when the combine in a perfect storm that nobody foresaw things fall over.

But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success.

100% of all devs make huge mistakes, at least once.

> But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes.

I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues.

Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite.

In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.

N+1 issues aren't nearly as devastating as N^2. Commend them for not putting your systems to a complete halt, then teach them how to reason about this properly.

>> a quality gate

Yes, this, also.

> N+1 issues aren't nearly as devastating as N^2.

Depends on the project, i guess: if you're unlucky enough to be working on a monolith and suddenly a page takes 5'000 SQL queries to load as opposed to 100, because someone thought that initializing data through service/DB calls in a loop is "easier" than writing views in the DB, it might still kill the entire system anyways, depending on the count of users.

And once this data initialization is sufficiently complicated and convoluted for you not to be able to rewrite it and them not wanting to rewrite it, all while "the business" is breathing down on your neck, you might either want to introduce caching (and possibly run into cache invalidation problems down the road), or just freshen up your CV.

I guess i'd also like to expand on the previous suggestion and advise others to consider performance/load testing as well, especially when coupled with APM solutions like Skywalking or even Matomo analytics, both of which can allow you to aggregate the historical page load times, CPM and overall performance of your applications, to figure out what went wrong when.

Still, that engineer (if its engineer's fault) is extremely unlikely to make that issue again. IMHO, the problem is systemic in that why the system allows such errors (if its human error) to happen. Given Google's scale I think a lot of the generally known scenarios are covered and what you see is tens of services interacting in not obvious ways. Those unobvious scenarios manifest in situations like this.