There is a tool that will do almost anything, but the tools are often just a bit janky. Rather than thinking of the toolchain as some polished, tightly integrated, perfect system, think of it more like an internal open-source ecosystem of things that work together but which are made by individuals, with limited resources, that still feature bugs.
This is not to downplay what we have, it's amazing in many ways, but most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them that cover almost every aspect of development. So much of the time I run into neat little systems and then discover they're a few Python scripts running under someone's personal quota. Sometimes these are 20% time projects, sometimes they are bits of tooling teams build for their own needs, sometimes they are more concerted efforts.
There is a core, tightly integrated set of systems that work well and are more "productized", but most of those aren't that much more special than what you can buy at any other company. If you've used Datadog then you'd feel at home in much of the modern infra tooling. Spanner is great, but anyone can pay to use it. Our CDN is very good, but there are other CDNs and from a developer perspective there isn't a whole lot of difference.
That's true at Google, just as it's true anywhere. But what makes Google a much better experience than most places is a willingness (and ability) to pay for internal dev tooling of every sort with dedicated headcount. And that results in a consistently better experience than pretty much anywhere (I understand Meta has a comparable culture and ability to fund it, and I wouldn't be surprised if some or many of their internal tools surpass Google's). It's also unfortunately not replicable as a strategy, because you need a giant money spigot to make it work.
If elsewhere, I think there are 2 kinds of tool: those developed by individuals/small groups in informal ways, some of these are bad, but much like with open-source tools, the ones that are good survive and become popular. The other type is those with more formal backing, dedicated dev/design/UX/PM resources, and those are good at surveying users to figure out what the right problem to solve is, so they're normally pretty good, and when they're not, they still accept fixes (I regularly contribute fixes to other teams codebases when I find issues).
I would be curious as to how you view projects that are staffed entirely 20%.
Why? Why reinvent the wheel 10 times?
There are systems to resist this and people whose whole job is to focus on unifying systems, but they aren't perfect. In general, I'd say that Google is way way way more uniform than most other companies because of the monorepo, centralized tooling, and (mostly) uniform development process across the company.
But this only solves a visibility issue (and codesearch/moma is already pretty good at this). It doesn't help you when the tool supports a different language or the team that owns it isn't willing to commit to the SLO that you need or whatever.
But even when there's a similar open-source tool, it may be difficult to integrate with internal systems. There's a lot of benefit to a consistent set of technologies (data formats, storage systems, etc) so that's often a reason to write internally.
There is a lot of open source software in use at Google so I wouldn't say there's a lot of reinventing the wheel. And when there is an internal clone, the requirements are often necessarily different enough to warrant writing an internal version.
I found the same thing moving from a very large company to a smallish company. Both had good tooling, the latter just had significantly less and there were gaps in coverage. Whenever I encountered one of those areas I often found myself wishing I had access to some niche tool I had gotten used to at my prior company.
In 2010 when I interned there, one of my mentors said something to me that stuck with me: "Google found a hose that money pours out of, and it's name is 'online advertising'. All we do is optimize that hose, and search for another one."
Sonos: https://www.theverge.com/2023/5/26/23739273/google-sonos-sma...
A MIT student: https://news.ycombinator.com/item?id=18566929
I met them (they were horribly unprofessional) in 2013 around time Sonos did to discuss my audio syncing tech. Best to steal tech then bother investing into R&D. Quickest way to make more money .. steal .. spend less and make more.
I haven't mentioned my experience in years cause it's old news but Sonos court victory is new relevant news to my experience meeting Google R&D that same year.
Though I can say for sure and claim is that they were extremely and laughably unprofessional to me and I met with other tech companies then who were very professional and polite.
Yet in the last decade, they really haven't had many successes (perhaps with the exception of Google Photos - 2015)
One would imagine that with nearly 200,000 employees at least one of them would have a good enough idea for a new product people like. But management and culture, with 'more wood behind fewer arrows', is no longer conducive to the good ideas getting launched.
I would argue that taking a product from 100m to 1bn users is a whole new level of success, and that has been done multiple times in the last decade.
Compute scales with the userbase. Engineering scales with the product featureset. That means a small userbase in general cannot support much engineering and therefore cannot have much complexity/features if it wants to be profitable.
That in turn puts big companies at a massive advantage. And in fact we see that - the vast majority of my time using the web is spent on big products (google, youtube, reddit, twitter), and only a small fraction on small sites (perhaps HN being the one exception here). Those big products won the battle for my attention.
And I'm not talking the UX, which has already changed substantially, but I mean the underpinnings.
There have been revolutions in apps. For example, pre 2017 I couldn't just tap a button and get an Uber. Or tap a button and have food delivered. Or do a video call with all my friends.
But the OS can't really claim credit for most of that innovation.
Not visibly. But the tech stack has changed substantially.
I'd find it quite an achievement if the UX has largely stayed the same while things under the hood got modernized over and over again and went with the times, subtly bringing innovations to the UX as well without people realizing. It's a feature that things don't look substantially different every two years.
https://appleinsider.com/articles/23/05/23/carplay-android-a...
> A report from Straits Research found that 98% of newly produced vehicles were compatible with either CarPlay or Android Auto. Meanwhile, 80% of prospective car buyers strongly preferred having these smartphone-based infotainment systems in their new vehicles.
The same research also shows Asia-Pacific to be the biggest market for these products, though North America is the fastest-growing.
Tbf. 25% connected is huge. If you just jump into your car to quickly do groceries or pick up your kids, you may not be interested in connecting your phone to your car, even if you really like that feature. So there could be a natural ceiling for the "connected" number and 25% feels getting close to that ceiling actually.
And everyone is just frustrated with laggy UI's in cars. But in reality they'll probably still just use Waze with a 10 buck phoneholder suction-cupped onto the windscreen.
People may have different preferences from yours.
Nevertheless, some people prefer not to accept them while not being entirely anti-tech either. You may not be agreeing with choices other people make, but it goes a little far to call them "absurd on its face", don't you think?
Not trying to downplay what happened after this: it was incremental change that made the products scale and be more usable to more people. I would say the things you describe as revolutions were actually evolutions of earlier products.
Which is somewhat offset by Google dropping Picasa.
I don't think this kind of analysis is really giving a good answer. I mean, by these standards, look what a disaster of a dinosaur legacy vendor Apple Computer is! Haven't had a successful product launch since the iPad 13 years ago! (Perhaps with the exception of Apple Watch - 2015).
TPU
Colab
Tensorflow
hmmm.
https://translatepress.com/is-google-translate-correct/
Another breakthrough is picture recognition. Did you know you can type in "motorcycles" and do an image search and there is ZERO text associated with many of those pictures of motorcycles? They really are mapreducing those pictures and running image recognition against a 17M open-source image library to assess "motorcycle-ness" of all those pictures! They started talking about picture recognition in the early 2000's but it really began happening in the late 2010's after they hired fei fei li, the mother of image recognition ...
Going neural definitely made a big difference but Translate was called "statml" by the infrastructure internally for years, maybe still is, because just using statistics and having training at all was a big deal back then.
Maybe it works well inside Google, but definitely not in the real world.
Arguably it was a bit thin on documentation but it just worked.
I think it's just something that happens to companies that get a certain size. The Process becomes more important than the Product.
When I was younger, I wanted so badly, to work at Apple.
A few years ago, they actually approached me, and I found out that the culture seems to have drastically changed.
They ended up not wanting me anyway, so maybe it's just sour grapes, on my part.
Can you explain in detail?
But I don't think that someone like me would be very welcome there.
That is both good and bad. I am certainly not God's Gift to Programming, so they may well be better off without folks like me. They certainly seem to be making a lot of green.
Wouldn't it be helpful to other people who are interested in working there?
This is a professional venue, and I don’t feel that slagging companies is a particularly professional thing to do. I like Apple, and wish them the best. I have no interest in throwing shade on them.
When I do say less-than-positive things, I find it best to stay vague, so people can easily write off what I say. I think that folks here, are perfectly capable of making their own decisions, and that I’m best off, letting them do so, unencumbered by my opinions.
But the software side? There seems to be a massive variance in both ability, and giving-a-shit across the software orgs, and no leadership filter on the output of these disparate groups.
Perhaps they have the opposite of the resting-on-laurels problem: a smattering of very senior people who are talented and care about the product/users, and a wide base of careerists who aren't very good and just want the money and the CV, users be damned.
I was told about e.g. multiple internal CI/build systems because one org didn't want to rely on the other, so they both built their own little software stacks.
That's a bit different than what I've heard from meta/google/amazon who seem to be more open to running centralized internal services
https://gist.github.com/chitchcock/1281611
This was rather a long time ago now so I wouldn't be surprised if things have changed since then, though having never worked at Amazon I wouldn't know one way or the other.
The obvious thing to do then is to not change too many things around it. Or at least document the changes.
So then you get a bunch of processes that are designed to not rock the boat too much, in case the magic dies.
I don't know if the process is more important, but at a certain size codified process becomes a requirement.
> I think it's just something that happens to companies that get a certain size. The Process becomes more important than the Product.
Why change it if it's working well? Keep doing what you are good at, keep being the best in the world/market at this, and the customers will reward you.
I seriously don't understand the common theme here in these threads saying that Google is basically stagnating. Why do they need to churn out a major new product every other year to stay relevant? They continuously re-vamp their entire stack, their data centers, their networks. Nothing ever stays as it is. As a user I actually find that things change too much. (But I'm not what counts, it's ad revenue that counts..)
Both can be true at the same time, and my feeling is that they are, having worked there. The business is not very diversified, and the tools are great.
That wasn't the argument. The argument was simply that having a profitable monopoly doesn't imply the presence of quality.
Google's massive internal processes and systems are the result of having already been huge for many years. Google needed the time and resources to build them up. They're not the cause of Google's becoming huge in the first place.
Of course the quality of Google's external search results helped Google achieve a monopoly. Ironically, many people say that Google's search results have been getting worse, and I'd have to agree with that sentiment.
See also the list of mergers and acquisitions by Alphabet: https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitio...
I've give one example I observed commonly early in my career. The unexplained memory leak. Some process is running and the memory usage continues to grow. Eventually it will use the entire memory of its machine and die. You have a few options: 1) debug the issue and address the root cause 2) debug the issue and workaround it in some way 3) Give up and rewrite the code using some other kind of tooling 4) wake up people when the process dies and have them restart it 5) write a cron-job that restarts the process periodically.
What is the right answer? The best from a QA perspective is probably to identify the root cause and fix the underlying issue. Tools like valgrind have made this much easier in recent years, but it still can be a challenge. Pragmatically, my own answer (speaking generally, there are more different cases for different contexts) would be to time-box and investigation and fix, and if that wasn't achievable in reasonable time, just write the cron-job and work on the next problem. You can imagine very successful operations filled with kludges like that. Is that low quality?
But I can't think of anything like that in the last 15 years.
This is a very simplistic notion of what enables the creation of killer products. That has much more to do with understanding users' needs and identifying market opportunities. Good code writing processes are about code maintenance and scaling engineering effort, not dreaming up the next killer app.
I'll retreat to saying that claims about the superiority of Google's code writing process remain unsubstantiated.
During my 3 years at Google, I observed little brilliance. Not that my little corner of the giant org is a statistically valid sample.
I did see a fair amount of "Google is The Best!" sentiments and rejection of anything invented outside the company.
- The risk aversion to anything that threatens a big existing successful product.
- The product/feature approval processes that implements the above.
- The concern about launching something experimental or half-baked under the brand (vs a startup which does that by default).
Personally, I saw a lot of product and engineering creativity, but it was often stifled or watered down by the above.
I had a front-row seat to some really revolutionary ideas in Google getting to the prototype stage before being squashed in the gears of "We're chasing a market and that's not how users see this product working." Stuff where, if it caught on, it'd be a paradigm shift... But it turns out users don't want every paradigm shift that comes down the lane.
Because Google has (traditionally; this has changed in recent years) a real push-pull in authority between management and engineering leadership, the company can't commit fully to building a quality implementation of a status quo. Nor can it commit fully to chasing entirely new ways of doing things that could shake up an established market. In general, this... Actually kind of works out fine for them, more fine than critics often realize, because neither of those answers are always correct. Sometimes you get Gmail. Sometimes you get Google Drive. And sometimes you get iGoogle or Wave. And sometimes you get the stuff in between, like Reader or App Engine (really popular among the users, but the users don't have the money to make it profitable to commit to it).
It means that you wouldn't expect them to have products that fail because of unresolvable tech problems. You can see plenty of cases like Stadia where the tech was solid but the product strategy and leadership follow through was garbage.
N.b., I’m not saying nosefrog is this person, or even everyone at google or ex-google is like this. This was just my experience working with senior principal or other very senior engineering staff.
Generally programmers are Google are better than programmers I’ve worked with outside of the company. But not by leaps and bounds. And in my opinion it doesn’t matter as your pace of development is so much slower it matters little how good of a programmer you are.
My own experience at FAANG and non-FAANG is that __some engineers__ at FAANG get to dream big and demonstrate ingenuity, whereas at non-FAANG (especially startups I've been at) everyone can do this, though not all take the opportunity.
At my last FAANG where I was a manager, my opinion is that performance management and the focus on bullet points, metrics, and peer feedback for review time is a huge limiter for everyone else. I spent a lot of time creating cover for people on my team to actually try something new and innovative. More senior management constantly wanted to know why person X on my team was working on a new thing instead of trying to move some incremental metric somewhere else. Based on my experience during performance reviews for senior ICs, risk taking wasn't particularly encouraged at L6 (or even L7) and below as the cost of failure was substantial.
There is one underlying technology special sauce they refined to an art form, which is building a reliable system on incredibly unreliable components. This comes from their inception when the two guys building out the company knew how to write code but not build custom machines; their initial attempts at racks (hard drives and motherboards mounted to flexible plywood) are hilarious-looking by modern standards and broke down all the time, so they had to learn how to write software that was fault-tolerant at every link in the chain. That resulted in building an infrastructure that was extremely supportive of experimentation (if the system can survive half of it disappearing, it can survive a software bug crashing half the machines), and the rest of the shape of the company kind of flows from that.
It is worth repeating a lot that most companies do not operate at the scale or constraints Google does, and their approach to engineering is emphatically not one-size-fits-all.
Going to (eventually) mostly dependable simple servers with no redundancy was a big leap.
One high-ticket item relied on a huge Oracle database. Every other aspect of that product was patch-on-the-fly, silent-release new versions, forward-and-backwards compatibility of software... But the whole product had regular scheduled outages on the weekends because the Oracle part could not be patched on the fly, had a rigid schema baked into its relational architecture, and had to be brought offline for updates and migrations.
It was like everyone was zipping around in racecars and then they all had to pause for Grandpa to cross the street.
From an outsiders, it looks like all of FAANG has scaled to the point where delivery, perf and managing the business has become quite cumbersome. This, I suspect is why they're all used the economic climate to cut back, as they largely are unable to see the benefit of the additional hiring.
I have no doubt most of the talent is top tier, but being a strong engineer doesn't deliver profits nor cost savings on it's own.
Is this really the case? In high school I sure thought Google was a magical software heaven us mortals could but dream of working for, but now (and increasingly as of late) I'm strongly under the impression that 20 years ago they discovered unobtanium in the form of a very under-served search and ads market, they plopped down camp on top of it and turned it into a giant money firehose, and now to this day are simply operating that ridiculously high-margin business while trying desperately to find another vein of unobtanium because when you already have one, that's the only thing that even registers on your radar.
"Revenue from Google Network ads hit $7.5 billion (10.7%), and YouTube ad revenues registered at $6.7 billion (9.6%). In other words, at $54.5 billion, the total ad revenue from Google constituted 78.2% of the company's total revenue in the quarter." https://www.oberlo.com/statistics/how-does-google-make-money
78% between google and youtube. And given youtube isn't nearly as high-margin as the search product (can you even imagine the cost related to essentially storing the video history of the world?), I'm hesitant to say youtube is their 2nd vein.
So yeah. Turns out when you have essentially a monopoly in a high-margin business, you can do basically anything and build an entire private software ecosystem without going bankrupt.
I interpret "how google does it" to simply mean "look at all this cool stuff you can do when your product is this high-margin".