Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story:

Uber reinvented a ton of technology because of the idea that off-the-shelf solutions wouldn’t work at “Uber scale”. However, there was very little accountability for whether the in-house solutions were necessary, and working on these tools would get you promoted. So for every “Uber scale” problem that a team actually solved, there were a couple other projects that were just half-baked alternatives to the off-the-shelf software that they should be using.

It turns out that “Uber scale” is not really that large, despite the name. But engineers kept repeating “Uber scale” and building infrastructure.

The same problem occurs at the larger tech companies like Google, Facebook, Microsoft, Amazon, and Apple, but in different ways and to different degrees. And to a large extent, engineers are copying what other companies do, and bringing ideas from one company to another when they hang out after work or switch jobs. For example, you can bet that these companies mostly have their own containerization and scheduling systems, many of which are undoubtedly not competitive with Docker or K8s in 2020, but K8s only goes back to 2014 and all these companies are older. I’m sure Borg and Tupperware are great if you work at Google or Facebook but I’m also sure that they’re missing a bunch of tooling that you’re used to. Same thing with build systems. Bazel, Buck, Pants, Please, and that Frankensteined system that Chrome uses are all copies of each other but Bazel is the only one with a decent size community and ecosystem, as far as I can tell.

Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available.

> Same thing with build systems. Bazel, Buck, Pants, Please,

It's worth noting that none of these were (really) NIH. Blaze wasn't open source originally, and Buck, Pants, and Please were all essentially reimplementations of the closed source Blaze rebuilt by xooglers who went on to work at FB, Twitter, and presumably thought machine, though IDK if it's the same.

Then/concurrently, Google open sourced Bazel, which is mostly-blaze.

That’s absolutely right, but I think it illustrates one of the problems with inventing your own tech stack, is that it might make perfect sense when you start the project, but several years down the road it might just be an inferior version of an open-source project.

My personal sense is that this even applies to stuff like Borg, but who knows?

You can't really predict the future if a large tech company will properly fund an open source project to reproduce what they have internally, and if that will be any good in time, especially if none of it exists. In the mean time, you need a solution to problem X.

I've even written a few libraries like that, and still stuff out there isn't that good or does not exist years later, because it's too niche, yet still a generic issue in the industry. Can't open source it although.

In an alternative timeline, Buck for example could of been the OSS solution that google decided to devote their resources to, and then in the end it would of been a good idea that FB created their own version of it.

To be clear, what I’m saying is that “this is a problem with reinventing stuff”, I’m not saying “you shouldn’t reinvent stuff.”

You think Borg might be an inferior version of Kubernetes?

It will be.

Mind you, Kubernetes was originally made written at Google by engineers that previously worked on Borg. So they will naturally design it to address problems with Borg. Kubernetes’s original project code name was “Seven of Nine”, because Seven of Nine is a friendlier Borg.

Just because someone subsequently writes another program, that does not assure the superiority of the later edition. People who wrote Borg thought Omega was going to be better, but Omega failed except for a few things that got cherrypicked back into Borg.

I particularly can't agree that k8s is a necessarily friendlier Borg. Borg is on rails; there are fewer choices for the user to make. K8s tries to be flexible enough to be used in various production environments, which leads to it being harder to use rather than easier.

Some of the flexibility decisions for k8s come from Omega and are mentioned as impossible to backport to Borg.

Sometimes you also get issues on k8s where a google engineer responds "it's not done yet because we really, really don't want to repeat what a fiasco it was under Borg".

So I'd say it might be better even in direct comparison ;)

I imagine part of the perceived usability problem comes from the fact that often users of k8s are also playing the role of borg-sre, whereas 99% of borg users do not need to deal with that.

Sure, and to be clear, when I say “it will be” that it’s little more than a guess on my part.

Google has enough project churn internally that services and programs do stand a good chance of getting replaced, though. There’s plenty of history.

It seems like you might be under the impression that bazel and blaze are not the same codebase?

If Google made a good build system with features not available in open source, eventually open-sourced it, continue to use it, and lots of other people are eager to use it, how does that demonstrate that they shouldn't have invented a build system?

I guess the comment wasn’t clear enough, because that’s absolutely not what I was saying.

“Problem with X” is not the same thing as “you should not do X”.

I’m also not getting why you think that I’m ignorant of Bazel’s origins, specifically, since I was talking about Borg.

You'd replied to a comment about build systems and bazel.

Regarding Borg vs k8s, it's worth noting that k8s was never built to replace Borg or to even try to match the feature set of Borg. There are some nice things about k8s (like the ease of running an entire cluster on your local machine), but since k8s tries to be all things to all people it's not even close to being viable (or competitive with what Borg can do) for Google itself.

"Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available."

I agree, but I'd also add that the state of the world in 2016 when most of these infra projects were launched is very different from the state of the world in 2020. Back then, uber was not easily able to run on a bunch of OSS/cloud native solutions, but now for sure its not just possible but likely the best/most responsible way to do it [source: worked there for 4+ years]

One of the most satisfying moves of my career was moving from a FAANG to a "cloud native" series C startup. The "not built here" syndrome is very real at the FAANG's.

I will defend the FAANGs just a little bit-- at least Amazon, Google and Microsoft-- in that they are now large SAAS providers and so "building it here" is essentially the same as eating your own dogfood, which is very much a good thing.

I would also add that everyone is evaluating the tech stack of these companies in today's context. Google was founded in 1998. Facebook was founded in 2004. Uber was founded in 2009. Microsoft was founded in 1975.

Amazon EC2 was first offered in 2006 and was super new and immature for a number of years. Kubernetes was first released in 2014. Mesos was a research idea for a number of years starting in 2009 and didn't reach version 1 until mid 2016. Etc. Etc.

These companies generally invent their own solutions because the solutions everyone here thinks they should be using didn't exist or were not stable at the time they had a problem they needed to solve.

By the time those new solutions exist and are stable enough, it requires quite a bit of investment to migrate to the newer solution as these companies already have much of their tech stack stable and bringing in revenue on their in-house solutions.

Rebuild $infra from large company and provide it as SaaS was a pretty good recipe for a decent software business for a while. It might still be. For example, PagerDuty came out of monitoring systems at Amazon.

I feel like what's pointed out in this comment is so obvious that it shouldn't be needed ... but evidently it is.

Sadly, it does need to be pointed out, pretty much every single time. Same with the "Why does company X need so many employees?" question, which you can just reply to with this link: https://danluu.com/sounds-easy/

It's also a risk mitigation strategy. As your company's size grows, so does the liability of your dependencies that you don't control. If you can throw a few engineers at a problem and save yourself tens of millions of dollars in risk, it's worth it.

It’s not risk mitigation, more a risk swap between “unknown risk”—we don’t know if this third-party product is going to meet our future needs—and “known risk”—we have full control while we are willing to write checks to fund it.

In the latter case there’s still plenty of risk: you might be spending valuable time building something that doesn’t work, is obsoleted by future change or future products; and the one that most people dismiss, you will be at the mercy of its creators. The people who build your critical technologies will have power over the organization you might regret later, and if they do leave, finding replacements can be hard and/or expensive.

The net risk reduction of course depends on specific situations, it is not always better and it is not always a "swap". In some cases, eliminating unknowns reduces more risk than is added by building in-house, which is when it makes sense to build in-house.

Spot on! Never thought about it from this perspective. :-)

Hard to say. I'm sure that accounts for some of it, but at a certain scale, having a system be tailor-made to fit your requirements and workflows has a lot of value. And if you tend to be towards the leading edge of scale and tech, the tools that you should be using don't exist yet or aren't known at the time without hindsight. My shop wasn't even that large, and we built our own k8s because we needed it. Looking back historically, it was even available at the time, but definitely not as the robust and popular standard that it is today.

And once you have a whole system built, you're really going to have a hard time justifying re-doing your whole architecture to move to a product that does the same thing. Basically that only happens when you have a new head of engineering who comes in and goes, i don't care how many engineers you have to throw at it, and i don't care how much custom functionality you depend on that you lose, just figure how to replicate everything you do today using k8s.

This is revealed if you go to any of their tech talks too.

That's not to discount the engineers working there, they indeed do have a number of bright engineers. But the no accountability comment is spot on.

I work at a huge non-tech company (bank), and we are guilty of the same. Tons of stuff built in-house for stupid reasons, including an entire buggy IDE that people curse every day.

I have not had the privilege of working for a "tech company" where tech and tech employees are first class citizens. I'm pretty certain the in-house tooling would be much better at a company like Uber or FAANG than a company like mine.

I interned at a FAANG last summer (Google) and I was blown away at how good the in-house tooling was (including the in-house IDE).

Everything was seamless - I'd log in, tap my security key, and instantly have access to pretty much the entire monorepo and the rest of the production + deploy systems. All of the internal tools integrated with each other -- for instance, I could create a CL (i.e. a pull request) and then fix issues raised by the CI system, entirely from the IDE.

"Owning the stack" completely in-house also extended to hardware -- all of my builds happened inside Google datacenters too (not on my machine), and the development box they provided me was a Chromebook.

Nice to see a comment like this, I feel the same way. Google is the exception to all these other comments. The tools available to Google engineers make working outside Google feel painful and inefficient by comparison even with their unfettered access to all the third-party tooling.

It’s so good that it would actually be a factor if I were ever considering working somewhere else. Getting to work with Google’s in-house dev tooling is probably worth 15-20k to me. A lot more than 20k if the other company has a reputation for horrible tooling.

A company isn't going to pay you more to be less productive.

To fix the tooling? Maybe..

This is in contrast to Apple's internal software systems, at least if recent reports seem to be accurate; aside from (I assume Xcode), it seems the business-ey enterprise stuff is outsourced to contractors and software consulting firms and is a giant mess...

Granted, their DNA is still in hardware/devices/OS level stuff versus "services" despite all they claim to be, but how much more quickly could they move if they "owned more" of their stack?

I’ve heard similar stories about working at Sun back in the day. It was magical, supposedly.

Sun didn't have a monorepo -- at least not 1998 through the acquisition. Each org of a given size (Solaris os/net, install, etc) had its own separate island of source with its own build and test systems.

I guess my comment wasn’t clear. Talking about how you could just grab a random terminal with your smart card at Sun, and get work done. True at Google, true at Sun back in the day, not true at most places.

Yeah, stuff like this is one of my chief reasons I want to start working at a "top tech company". Just an overall higher level of tech competency.

You don't want to?

Typo, lol. Should have been "start". Or I may have been typing "stop working at a non-tech company" before my train of thought ran away.

Ah, I remember when the in-house IDE first came out, and it was considered a toy compared to vim/emacs. It has come a long way since then.

That said, I'm also looking forward to "VSCode front-end in the browser" becoming ubiquitous. I remember using Visual Studio back in my first job in high school, and Intellisense made coding so much more... explorable and approachable.

Which bank if I may ask?

Also worked at one that developed a ton of tools internally, but I wouldn't call many of them terrible.

I'd rather not mention specifics, but suffice it to say it's one of the big US bulge bracket investment banks.

I think Uber's finally recognizing this and has been adopting more off-the-shelf stuff for non-core stuff. The move to slack a few months ago comes to mind.

In the case of those larger orgs though, were the off the shelf solution was mature enough for them to use at the time?

Same thing at Postmates

- nothing wrong with NIH syndrome if you have the means and resources to refine existing ideas and rebuild them with scale in mind

- uber is a 50 billion dollar company and they have to de-risk themselves by owning entire stacks, top to bottom. if it means re-creating something from scratch... who cares, they have billions.

While I understand the point that they have money to burn, there are better ways to spend that money. Some items like “databases” are core to their SLA that it makes sense to invest. Build, source code management, etc are not “critical to de-risk”. It is not like Microsoft will one day refuse to sell them GitHub enterprise or GitHub actions because some division of Microsoft is competing with them.

That’s just apologia for NIH syndrome. It’s fine to reinvent software if that serves your goals. It’s short-sighted to reinvent software without due consideration for whether off-the-shelf solutions work. That’s not de-risking, that’s adding risk and slowing development. You see it happen at companies like that and usually it’s a sign that there’s something wrong with the culture, incentives, and the way that promotions work.

My take on it is that the engineers want to make complicated solutions to hard problems to justify their salaries and get promoted, and that managers encourage that behavior so they can defend their headcount. I’m not accusing any of these people of acting in bad faith here—nobody’s reinventing tech to sabotage the company, it’s just that the system encourages this kind of behavior.

This problem is not unique to Uber, you’ll see similar things happen across the industry to different degrees.

Well, the "due consideration" needs to duly consider the actual costs and benefits of the off-the-shelf software, not make the kind of unsupported blanket claims about its obvious superiority, like you just did. Every off-the-shelf program has maintenance costs, flaws, and weak fitness for purpose. You just made a sweeping and italicized claim that all off-the-shelf software is both less risky and faster to develop than house-made software. That's a dangerous philosophy and probably isn't true for every organization and application.

> …not make the kind of unsupported blanket claims about its obvious superiority, like you just did…

Not what I said. Let’s move on.

> That's a dangerous philosophy…

Go pick a fight with someone else.

Or, at that point, you use industry standard solutions and hire the best subject matter experts on that solution. Like, for example, using Cockroach DB and hire one of the core committers cause you're big enough. It's what MSFT did with things like the Python extension for VSCode and DBOX did by hiring Guido since they're a Python shop.

I still shake my head whenever I hear about how Uber built an entire Slack clone for "Uber-scale".

My experience working at Facebook was that the "working at scale" is nowhere near as glamorous as it sounds, especially when it comes to scaling build systems and monorepos.

As a company, you have to invest immense resources in evolving the infrastructure and it is a never-ending task: the more capacity you build, the more your organization will grow in ways that tax that capacity. The goal posts are ever-receding.

As an individual developer, all that fancy distributed infrastructure becomes a barrier that must be laboriously overcome at every step. Your expectations about what constitutes a "fast" operation get gradually distorted. Working on "small" side-projects (a few thousand files or less) feels effortless in comparison.

Not to say you shouldn't seek out the "big tech" experience; just that it is not all roses.

Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings

> Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings

And a lot of tearing your hair out when trying to use tools in the ways that they weren't intended to be used.

Of all languages, Go is not the language where you want to be doing things against the grain (that is, against the way the language designers intend it to be used). Some people like that about Go, and some people don't. Either way, Go is a very opinionated language with a very opinionated ecosystem, and ignoring those is a recipe for frustration.

Last time I checked, Bazel was not recommended by the Go developers - and with good reason: there are a lot of gaps in rules_go/Gazelle, which this blog post alludes to but glosses over. While Google uses Blaze internally, Blaze is not Bazel, and the differences are very apparent to anyone who has used both to build Go specifically. Furthermore, rules_go was developed entirely independently of the Blaze ruleset that Google uses internally, so it's a tool that's not actually used internally at Google, but also not used by the majority of the non-Google Go developer community either, in addition to not being recommended by the Go team at Google[0].

[0] There is exactly one mention of Bazel on the entire golang.org domain - in a changelog from over two years ago, in an /x/ package, where Bazel is mentioned as one of two build systems in a "such as" clause that the new package could potentially enable support for (/x/ packages are considered experimental and not subject to the same backwards compatibility or maintenance guarantees as the rest of the project).

Having used both for years I honestly have not perceived the difference between blaze and bazel go_binary/go_library/go_test. What's the big difference?

From the issues I've read on github, anecdotally, is that the approach of the gazelle and rules_go teams is to leverage the standard go tooling and not make changes to go itself.

Also anecdotally, bazel and golang work really well together IME. The community seems pretty active, and the upsides of using gazelle/bazel with golang seem to outweigh any downsides (though I'd be hard pressed to name a downside, that isn't inherit to golang itself).

> Also anecdotally, bazel and golang work really well together IME... and the upsides of using gazelle/bazel with golang seem to outweigh any downsides

This is really not my experience from having used Bazel with Go for the last four years. But I'm happy you are are apparently not running into issues.

I've been using about the same amount of time. Have you tried using python and bazel? Now there's some real weeping and gnashing of teeth ;) Lot's of bad code depending on python == python2 type of non-sense.

Javascript is another target that is just about as gnarly. The node module resolution algorithm is reimplemented a million similar-but-not-quite-the-same-ways by popular packages, most of which struggle dealing w/ symlink-heavy codebases (like bazel sandboxes).

> Either way, Go is a very opinionated language with a very opinionated ecosystem

except when it comes to dependency management

We're not at the scale of Uber yet, but at Jupiter (YC S19) we're using tech like Bazel, protos, and Kotlin to prepare us for when we are: https://starship.jupiter.co/jupiter-stack/

We're actively hiring for a senior SWE right now, so feel free to shoot me a note if you're looking.

Protos is more of a faster & smaller network serialization tradeoff vs. ease of development thing than a company scale thing. And a thing to do if you want to integrate with gRPC.

I worked at a company that used protos from day one for example.

Many large companies are still using JSON w/ schemas as their network serialization layer just fine.

I'm not familiar with the Kotlin build chain, but what role does Bazel play? Is it needed for integration with the protocol buffers?

Yes - Bazel has excellent protobuf integration compared to other build tools (we were originally using Gradle). We're compiling Kotlin to the JVM, so the build chain is similar to Java's.

Gradle has a great protobuf plugin, what is so not excellent compared to Bazel?

This. It's very easy to forget about the political processes involved in making projects like this happen. Also, getting a job at a company as big as Uber in no way guarantees you'll be able to work on tech like this. You'd need to make sure you're joining a team close to this project, and if the culture of team turns out to be toxic, the fun of working on "cool" tech fades really, really fast.

Uber Engineering sounds like they are collecting Google's open source tools and name-dropping them in their blogs to market themselves as "cool tech".

One companies boring tool is another companies favourite hyped third-party tech tool. (Until it doesn't work for them)

Scale is not a prerequisite, but you do have to sell it to your company. If not a greenfield, porting a codebase over is not easy, and you have to do your organizational excellence homework too. Monorepo, well reviewed code, team not in permanent crunch, high-quality tooling culture.

Just like Kubernetes, Bazel is not a silver bullet. But if you get all your ducks in a row, switching over is a gamechanger and will help you out even at the smallest scale.

You want to join Uber because you think Bazel is cool tech?

Having spent the last few weeks of my life converting our build system to Bazel I guarantee you it's far from what I would consider cool. It's more of a necessary evil because the alternatives are even worse.

Not just large companies - plenty of small startups building cool developer tooling. And much more fun when you get to start on a green field, rather than slowly migrating the existing code over the course of a decade.

They just had a laid-off recently, and still not profitable given how much $$$ they have burnt over the years. I am not sure if you wanna work there.

FANG such as FB and Google has some big monorepos, and you can try that if you are interested.

More politics, then tech.

I wish they weren't morally bankrupt as I would have loved to work on their tech stack.

> morally bankrupt

How so, in 2020?

I think once you're stealing critics' medical records, your morally bankruptcy looks more like a chapter 7 than an 11. There's no expiry date; raze the whole thing.