> We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake.

But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that).

Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.

I've been confused for a while about the common claim that Bazel gives "full hermeticity" of builds -- it doesn't seem to be true in practice (at least for packages with system dependencies). Maybe you can help me clear it up.

E.g. Google's protobuf libraries [1] can be built with Bazel, and they depend heavily on system headers outside the repository, e.g. <iostream> and <stdio.h> and lots more. If those headers subsequently change, Bazel will not pick up on this and will not know to rebuild the parts of the build that depend on them.

To reproduce: run `bazel build -c opt //:protoc_lib` and then put random garbage in your /usr/include/stdio.h and /usr/include/c++/<version>/iostream and then rerun the bazel command -- it will not know to invalidate the build cache. If you `bazel clean` and then build again, you'll get different results.

Bazel does a lot of really nice things and I can believe that within a google3-like environment (where the source code never references a system header?), it effectively provides hermetic builds, but in practice as used outside Google (or even in Google's public OSS releases) it doesn't seem to really match this description or enforce a hermetic seal. What am I missing?

[1] https://github.com/protocolbuffers/protobuf

The best way to get your hermetic builds in order is to _always_ use remote execution and ensure that your executor nodes don't have compilers and headers just laying around. Force yourself to get the toolchain under bazel's control.

I think the reason the public finds this so mysterious is the documentation for CROSSTOOL is terrible, and virtually all of the people who learned how to use blaze at Google go out in the industry with no understanding at all of CROSSTOOL, because there's a small dedicated team who maintain it.

Does your company use remote execution across the board with Bazel? How much effort was it?

I work for no man, as a character in a movie once said. But I recently went through this exercise just to make sure I could do it, and it's easy enough to put a bare-bones executor node in the cloud and use it temporarily. You can even get ARM nodes, to make sure you aren't implying the target architecture.

Doing it on a company scale is probably harder. Last place I worked used bazel but did not bother with remote.

I wish Bazel was more explicit about things leaking in from outside the workspace, but you can always vendor the toolchain to be more hermetic. For example see https://github.com/grailbio/bazel-toolchain which sets up LLVM and Clang for you.

https://github.com/bazelbuild/bazel/issues/4558

Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf. (And not particular to Bazel -- a developer can make any build system effectively hermetic by vendoring everything.)

In my view the challenge here is that a dependency changes (e.g. /usr/include/stdio.h is upgraded by the system package manager, or two users sharing a cache have different versions of a system library) and Bazel doesn't realize that it needs to rebuild. It would be a pretty heavy hammer if the way to fix that requires every OSS project to include the whole user-space OS (C++ compiler, system headers, libraries) in the repo or via submodule and then be careful that no include path or any part of the build system accidentally references any header or library outside the repository.

And maybe this issue just doesn't need to be fixed (it's not like automake produces build rules that explicitly depend on system headers either!) -- my quibble was with the notion that Bazel, unlike CMake or whatever, provides fully hermetic builds, or tracks dependencies carefully enough to provide an organization-wide build cache across diversely configured/upgraded systems.

> Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf.

No, that's up to the user. If you include protobuf in your Bazel workspace and have a toolchain configured, protobuf will be built with that toolchain (this is also how you would cross-compile). Bazel workspaces are still a little rough around the edges and interactions between them can be very confusing, as Google doesn't need them internally (everything is vendored as part of the same tree).

Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.

Of course, the whole model falls apart if you want to depend on things outside of Bazel's control. However in practice I've found that writing Bazel build files for third-party dependencies isn't as hard as it seems, and provides tons of benefits as parent mentioned.

For example, last week I was tracking down memory corruption in an external dependency. With a simple `bazel run --config asan -c dbg //foo`, I had foo and all of its transitive dependencies re-compiled and running under AddressSanitizer within minutes. How long would that take with other build systems? Probably a solid week.

Completely agree that you can't compare Bazel and CMake directly, they are not on the same level. But for compiling C++ Bazel will have to compare itself to CMake, which is plain and simple the defacto standard.

All the complex things that Bazel brings into the game (distributed builds and caches primarily) simply don't have any value when the main task that you're trying to make faster is not done well.

I would much rather like to see more tooling that does these things but builds upon layers of battle-tested existing software.

> There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.

I think this is something that's slowly, but surely, changing. There's a lot of adoption of Bazel from some big players out there (Uber, Bloomberg, etc). The Blaze team @ Google has also started incorporating outside OSS maintainers into their OSS software (specifically rules_python).

Things are improving. Many of the things you've addressed (third party libraries, cross compiling, dynamically linking things) have been talked about by many people in Bazel's slack. It might not be flawless currently but Bazel's abstractions of how builds work are cool, RBE/caching is amazing for large projects, and a single build system with the promise of gracefully handling "every" language is something our field has been needing for some time.

In 3 to 10 years I can imagine being able to start writing code and never think about which language some library I want to use is written in, which package manager and toolchain I need to use to build everything, how to package my code to be deployable in production, how to setup unit tests, how to get autocompltions and docs and everything in my IDE, etc. The answer could just be "bazel, a language server, an IDE that speaks lsp".

CMake is a natural fit for C/C++ (even though I strongly dislike the syntax). It has a good IDE project generation, and you can use ninja to build super fast. But it isn’t pre-configured for hermetic builds out of the box, remote, team-wide builds and caching are much further than a command line option away.

You can use sccache with Cmake easily. On mobile so I can only give you breadcrumbs.

1. Set up sccache backing server (I use docker redis) 2. Export sccache config. Set SCCACHE_REDIS to a Redis url in format redis://[:<passwd>@]<hostname>[:port][/<db>]

Set CMAKE_<LANG>_COMPILER_LAUNCHER (cc, cxx) to sccache.

I think the biggest thing to remember is that changing your build tool for a large project is going to be a massive effort, you're gonna need a LOT of hours to throw at it, and you might not find many benefits for a long time.

Starting a new project with it, so it can help you do things the "right way" is a much more attractive proposition.

That said, the IDE/tooling situation for Bazel was disappointing to me in the Java world too. It definitely lags and is incomplete.

I was keeping tabs on the Python plugin for a couple of years (but stopped checking in about a year ago) and Python3 was advertised as "supported" the whole time, but no one was able to get it working. There were a number of open issues, none with any action on them.

And while Nix has an insane learning curve, figuring out how to extend Bazel (and consequently build one's own Python plugin) was a nightmare. The documentation was unhelpful/incomplete and the codebase was incomprehensible and poorly organized, with inheritance sprinkled on everything for its own sake (which is to say, par for the course for Java apps).

I know... "Don't look a gift horse in the mouth! / Don't complain about open source software!" Fair enough, but I don't have to use it, and I'm free to caution others not to as well.

One of my side projects was setting up a K8s cluster that ran Hydra in containers, used Hydra to build packages, and Kubenix to build and deploy services to the cluster. It was a little closed loop.. my goal was to have it building and deploying itself fully.

Unfortunately Kubenix was too immature, my knowledge of Nix insufficient, and Hydra required privileged containers due to Nix build sandboxing.

Have you used nix as a primary build system rather than a meta build system? If it works, I might have the energy to revive my project.

I'm not sure what you mean by "a meta build system" vs "a primary build system", but for many years Nix was what our company used to build and package our software. Unfortunately, it was a very bad experience; Nix doesn't strive to be a build system, it strives to be a package manager. The learning curve for writing packages is too steep to expect every engineer to learn it, and when things broke it was often hours and hours debugging into some obscure C library way down the dependency tree that no one in your organization has ever heard of. Worse, very little is documented well or accurately. Trying to figure out, for example, how Nix's python packaging libraries work is frustrating--partly because there is no obvious way to find the defining file for a given package dependency and partly because the package definitions in nixpkgs are rarely documented. These are all tractable problems, but Nix doesn't (seem to) aspire to solve them.

FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.

Bazel has extensive support for cross compilation. You can find pre-made toolchains. Intefacing with third party libs is as easy as writing a BUILD file.