News for hackers. Stuff that matters.
Re:Software Engineering at Google by hn_throwaway_99 on Monday August 14, 2023 @01:03PM (#37123494)
Strong disagree. In my experience, this is not commonly why competitors don't adopt Google's practices. The main reasons I've seen are:

1. Money. Google essentially has a giant, gargantuan, enormous, bottomless pit of money to build a lot of this tooling (and also to take the risk if something ends up not working out). I think you might be able to say that other companies are just being short sighted if they don't implement some of these things up front, and that may be true, but (a) that's pretty much human nature, and (b) given that very few other companies have a bottomless pit of money like Google, that may just end up being the right decision (i.e. survive now and deal with the pain later).

2. Talent. This is closely related to #1, but few other companies have the engineering talent that Google does. If there is one thing I've seen with my experience with ex-Googlers is that most of them are fast coders. So when you go to your boss and say "I'd like to implement engineering/tech-debt improvement XYZ", at other companies it's a harder decision if (on average) it would take 9 months to implement vs. 2 or 3.

3. Related to both of the above, but your 4th bullet point, "Quiet accretion, over several years, of the [[whatever]] by people who understand it.", is actually other companies just waiting for more evidence to see what "shakes out" as the industry-standard, optimal way to do things.

4. Finally, your stage 1, "We're not Google, we don't need [[whatever]]" is actually true in tons of cases. Many of Google's processes are there to handle enormous scale, both in terms of their application/data capacity, as well as the sheer number of engineers they need to coordinate. Very, very, very few companies will ever hit Google's scale.

Re:Software Engineering at Google by jeffbee on Monday August 14, 2023 @01:07PM (#37123547)
Just discussing your point #1, I hear this but what I see is that the companies I have direct experience with spend much more and move more slowly with their we-are-not-Google hacks. People move fast and break things into a corner where their entire project is a haunted graveyard with no test and no comments, that has never been reviewed, and at that point nobody is allowed to change anything.
Re:Software Engineering at Google by vbezhenar on Monday August 14, 2023 @01:22PM (#37123736)
Tests, comments and code reviews is not something unique to Google. It's a commonly accepted practice. There might be some dark corners, just like there are people who perform version controlling with ctrl+c, ctrl+v technique, but it's not a norm. I don't think that many people would argue against basic software development rules. However being Google is much more than writing tests and doing code reviews.

Being Google means having a team which writes source control management system for you.

Re:Software Engineering at Google by hn_throwaway_99 on Monday August 14, 2023 @01:21PM (#37123730)
Perhaps, but to me everything you put in your comment above just sounds like bad engineering practices in general and not something particularly related to Google processes.

E.g. things like "do feature work on a branch and then code review/run PR checks before merge", "have unit test coverage (being a hard balance to judge what is sufficient coverage)", "have useful comments" - absolutely none of these things I associate with "Google engineering practices", and many of them definitely predate things that were specifically done at Google.

Things I think of when I think about Google practices are things like ensuring data is infinitely horizontally scalable, monorepos, etc. Those things are all scale-specific.

Re:Software Engineering at Google by nostrademons on Monday August 14, 2023 @03:59PM (#37126053)
Monorepos actually work better at small scale than at Google scale. I think it's nuts that individual startup founders actually consider microservices; if you are validating out a software product idea, write a computer program, the very simplest one possible, to prove that you can do it and get the general shape of the architecture before you start dividing it into microservices.

I usually see the pressure to split into microservices appear around 20 engineers, just as your single repo is starting to get unwieldy. Knowing that the big companies use a monorepo is pretty important information here, because it may prompt you to invest in tooling to make that one repo less unwieldy rather than splitting into many small repos that will be very difficult if not impossible to merge back together again.

Google doesn't actually plan for infinite horizontal scalability in data. The framing I've found most useful is [Jeff] Dean's Law: "Plan for a 10x increase in scale, but accept that sometime before you reach 100x, you will have to rewrite the system with a different architecture." The reason for this is shifting bottlenecks: as the system gets larger, different aspects become the bottleneck to future scalability, and each time the bottleneck changes you usually need a different architecture. But by planning for an order of magnitude growth, you ensure that you're not artificially introducing bottlenecks, and that you have enough headroom to actually discover the new bottleneck.

Re:Software Engineering at Google by hn_throwaway_99 on Monday August 14, 2023 @04:43PM (#37126576)
Re: monorepos, I think we're talking about 2 different things. I usually hear the term "monorepo" discussed in the context of how it is practiced at places like Google and Facebook: having the code for all the company's services (micro or not) stored in a single source control repository.

A monorepo really doesn't have anything to do with how code components are deployed - your comment seemed to be contrasting a monolith architecture with a microservices one.

Re:Software Engineering at Google by nostrademons on Monday August 14, 2023 @09:20PM (#37128990)
I was fast & loose with terminology, but I'm thinking of the organizations where every binary and every library is its own Github repository, and you make copious use of git submodule to build anything. I think that's the same thing you're talking about, right?

It's impractical (particularly when the project is young) for the same reason having separate binaries is impractical: it makes it very difficult to do refactorings that cut across repos while still keeping atomic checkouts and rollbacks.

Re:Software Engineering at Google by hn_throwaway_99 on Tuesday August 15, 2023 @06:35PM (#37140162)
Yep, we're talking about similar things but in slightly different ways:

1. When you're small enough, and only have a few teams and/or deployed binaries, you keep everything in a single repo.

2. As you grow with more teams and more products/binaries, often companies will split into having separate repos per team/product, and then use some sort of dependency management tooling (either something like the git submodules you discuss, or a private package repository, e.g GitHub's Package Registry, Nexus, or Artifactory). I totally agree with you that a lot of companies do this prematurely.

3. What distinguishes "monorepos", in my opinion, is that this was an innovation at either Google or Facebook I think (not sure who was first), where they realized exactly what you point out - it makes it really hard to do refactorings across lots of dependent projects in different repos. So they decided to keep everything in a single source control repository, with a single commit history. But in order to do that and have things be sane with lots of teams and thousands of developers, they needed to invest a ton in custom tooling to make these giant monorepos usable by individual developers and teams. E.g Facebook has Sapling, Google has Piper, and there are open source tools like Lerna for JS monorepos.

So, in my experience, just having everything in a single repository but without any special tooling (because you're small enough to not need it yet) is just a repo. Monorepo IMO implies that you've grown to the point where it's difficult to keep everything in a single repo without special developer workflow support tools.

All that said, I definitely agree with your main point - a lot of companies can just keep everything in a single repository a lot longer than they think they can, even without special tooling beyond some separate build targets.

Re:Software Engineering at Google by ryandrake on Monday August 14, 2023 @08:15PM (#37128506)
Yea--I've found it surprisingly difficult to get plain vanilla medium-sized companies to adopt obvious, time-tested best practices. I'm not talking about "Google engineering practices" but basic table stakes practices like using source control and a bug tracker. So-called "Joel Test"[1] items. The most common excuses are: "We don't have time/money to do infrastructure/process, we need to write shipping code!" and the usual "We've always done it this way".

1: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

Re:Software Engineering at Google by hn_throwaway_99 on Tuesday August 15, 2023 @06:42PM (#37140245)
It is honestly baffling, totally baffling to me that in this day and age there are companies that don't use source control or a bug tracker (but trust me, I believe you).

With these companies I think it's just best to walk the other way. Some of them may state that "yeah, we know we need to improve our engineering practices, so we're willing to learn" but they usually just have so much of a different mindset of what it takes to actually run a software company that it's just a waste of time. There are many more companies that have a modicum of understanding about what they're doing.

Re:Software Engineering at Google by OkayPhysicist on Wednesday August 16, 2023 @12:49PM (#37149847)
These rarely are "software companies". They're companies in other industries, that happen to need some software. Sometimes it's a pretty plumb gig: Good, but not great, pay, often in relatively lower cost-of-living areas, a relatively light workload, a good amount of autonomy as one of a maybe a handful of software devs, and in their blindness to good practices (like source control), they're also untouched by common bad practices in software, like whatever bastardized version of agile/scrum that your bosses heard through an extremely lossy telephone game.

But there's also the bad: Software isn't the company's focus, so you aren't the company's focus. That means no "Senior FAANG" salaries, no best practices to keep things sane, and often you find yourself working on a codebase that was originally hacked together in a week by a chemist who may or may not have been deliberating huffing reagent fumes.

Re:Software Engineering at Google by majesticglue on Monday August 14, 2023 @03:36PM (#37125757)
clearly there is a missing link in your experience of companies where you don't have control over a lot of the things you think is needed and companies want to push a product as fast as possible and cut as many corners as possible.

More and more companies will be like this to cut costs

Re:Software Engineering at Google by choppaface on Monday August 14, 2023 @03:55PM (#37126008)
One conspicuous omission in the ex-google is reflection on killed products like Google Wave, Plus, Glass etc etc etc .. for many of the [whatever] was the gross imbalance between Eng owning the product but ignoring the userbase.

What ex-googlers often fail to grapple with is the product lifecycle (how short it may be) and the value of having diversity in the loop of product testing. Google is designed to be a safe place to focus, and that’s not what the real world is like outside the plex.

Re:Software Engineering at Google by jeffbee on Monday August 14, 2023 @04:29PM (#37126396)
Actually I think it is never-Googlers who have the wrong perspective here. The fact that Google constantly produces and destroys products demonstrates that it is extremely easy for that company to churn out code, and validates their software development methodology. It's incredibly easy to just dash off a product building on their gigantic foundation of source code, infrastructure, and launch process.

The fact that Plus and Glass got canceled and Wallet has been canceled sixteen different times is merely a consequence of the fact that leadership and product is often led by imbeciles. That's an organizational problem and I hope nobody is out there cargo-culting Google's org (even though I know they are, with OKRs and Perf being widely copied).

Re:Software Engineering at Google by choppaface on Monday August 14, 2023 @05:11PM (#37126839)
Exactly, Google engineers deflect user issues and product failures to the leadership and non-engineers. That happens in any large team, but Google has sweetened the situation for engineers to keep them focused on engineering rather than the larger consequences. E.g. credit cards are just fine, nobody actually wants to see ads, etc. It’s the user’s fault for failing to see the esoteric details behind the thing.
Re:Software Engineering at Google by nostrademons on Monday August 14, 2023 @04:02PM (#37126090)
Google's penchant for killing promising products is 100% the result of poor incentives. People are incentivized for launching challenging projects, but they are generally not responsible for the bottom line (which is going to be dwarfed by Search Ads revenue anyway) or for user happiness & brand loyalty (which is challenging to measure). As a result, lots of promising and exciting products are brought to market and then killed, as the easiest way to bring new products to market is to cannibalize the stuff your predecessors did and show how great your alternative is instead.
Re:Software Engineering at Google by hn_throwaway_99 on Monday August 14, 2023 @05:32PM (#37127084)
I'm not sure why your comment was previously downvoted. I've often heard, and it's not hard to find these comments from ex-Googlers on HN, that Google's "promotion-oriented development" is one of their biggest factors in some of their cultural shortcomings. That is, launching a big new product is seen as one of the best ways to get promoted, while working on the little nits (which in my experience, especially with some of Google's enterprise products, can languish for years, even though they can be really important but "boring" issues to fix) is not seen as high-value work.
Re:Software Engineering at Google by RichardCA on Monday August 14, 2023 @02:16PM (#37124512)
I think it's also the ability to have a deep bench of coding talent who just get to work on the toolchain. Most companies ration that talent to the product, shipping features that drive revenue.
Re:Software Engineering at Google by underdeserver on Monday August 14, 2023 @03:57PM (#37126032)
Eh, money is only a factor when it comes to scale. That is, Google can afford to hire 30 engineers to support their CI infra, you can't.

Everything else isn't. Unit tests aren't a luxury that Google's infinite riches allow it to have - they pay dividends whenever code exists for more than a few weeks.

You can bet your ass Google engineers don't write unit tests for throwaway code.

CI saves time, and while Google can maintain a team you can afford to pay for Jenkins or GitHub Actions, because not paying for them is more expensive - if your company is to survive for more than 3 months.

Re:Software Engineering at Google by hn_throwaway_99 on Monday August 14, 2023 @04:38PM (#37126515)
As I said in another comment, I think folks are just disagreeing on terms. I absolutely don't consider things like unit tests or CI to be any kind of "Google-specific" engineering advice - they're just standard good engineering practices.
Re:Software Engineering at Google by jeffbee on Monday August 14, 2023 @04:54PM (#37126688)
True, but what I was trying to introduce into the discussion was what another sibling commenter astutely labeled the anti-cargo-cult: industry feeling that anything at Google is an anti-pattern even when that thing is firmly established among other successful software developers for a long time. And in my experience comprehensive unit testing is one of those things that I have sometimes heard waved away.
Re:Software Engineering at Google by JonChesterfield on Monday August 14, 2023 @04:15PM (#37126226)
CI can totally cost time, especially if it doesn't have a team of good engineers keeping it running sanely and ensuring it tells you useful things on failures.

A CI bot which waits 24 hours then says "no", with a text file that crashes your browser and ultimately only contains the information 'exit code nonzero', which fails for reasons totally unrelated to your code change is dubious as a value add system.

If that bot is also a non-negotiable gate on shipping things you get a bunch of other antipatterns, like massive code patches to decrease how often you have to roll the die and a tendency to hit retry every day or so until the probability that it's actually your patch that's broken gets high enough that you try to debug it locally, at which point you may be unable to reproduce the blocking error anyway.

The real question is whether that pathologically rubbish implementation is still better than shipping without CI, which rather depends on whether your engineers ship code that works without the guide rails, which to a fair approximation they do not.

Thus it might still be a net win for product quality but saving time is harder to see.

Re:Software Engineering at Google by hn_throwaway_99 on Monday August 14, 2023 @07:35PM (#37128219)
Setting up a decent CI pipeline on GitHub with GitHub Actions is super, super easy - less than a day's work for a basic initial implementation.

Of course, the difficult part about managing a CI pipeline is writing quality tests, ensuring your tests don't take forever, deciding the right balance between mocking out 3rd party runtime service dependencies vs. calling their dev versions in your tests, etc.

But this is why I argue that the bare minimum should just be to have the CI pipeline created. If you don't have that, you are definitely going to screw over future you. Once that's there you can balance the cost/benefits of how much to invest in your test suite and test coverage.

Re:Software Engineering at Google by jabradoodle on Monday August 14, 2023 @04:04PM (#37126114)
I don't think only Google are writing unit tests.
Re:Software Engineering at Google by orochimaaru on Monday August 14, 2023 @07:19PM (#37128094)
I will add one more category - companies that have or need scale but nowhere near the technical talent that google has.

Eg - most telco companies in the US run at a scale similar to google. They need most of the software engineering best practices, internal tools teams, etc. They used to have it during Ma Bell times when they had a cash cow. That doesn’t exist and they’re left with scale and the points 1-4 described above.

This in general leads them to outsource to lowest bidder contracting firms that compound the shitty software problem. In the end it’s a miracle that all of it works together :)