News for hackers. Stuff that matters.
Re:When to use Bazel? by dietrichepp on Tuesday September 13, 2022 @04:12PM (#32829612)
I've been using Bazel for side projects these days, including small retro game projects for old game consoles. The entry price was high, but it works so well I have a hard time imagining working without it.

For retro game projects, the core of your game might be written in C or C++, which you want cross-compiled. That's easily within reach of stuff like makefiles. But then I start adding a bunch of custom tooling--I want to write tools that process sprites, audio, or 3D models. These days I tend to write those tools in Go. I'm also working with other people, and I want to be able to cross-compile these tools for Windows, even though I'm developing on Linux or macOS.

My Bazel repository will download the ARM toolchain automatically and do everything it needs to build the target. I don't really need to set up my development environment at all--I just need to install Bazel and a C compiler, and Bazel will handle the rest. I don't need to install anything else on my system, and the C compiler is only needed because I'm using protocol buffers (Bazel downloads and compiles protoc automatically).

Re:When to use Bazel? by synergy20 on Tuesday September 13, 2022 @04:37PM (#32829887)
Makefile can do all that too, without java dependency and straightforward to read at any time later, what real extra benefit Bazel brings in here.
Re:When to use Bazel? by yata69420 on Tuesday September 13, 2022 @05:24PM (#32830365)
The benefit is that it keeps a huge graph of all dependencies, so everything from the toolchain through your application can be built with hashing and caching. Sort of like make, but decentralized and based on hashed file contents.

Once you have bazel, you can distribute the workload so that you could have thousands of machines each producing artifacts to be shared with other build machines.

Then you can set up your dev machine to rely on those caches, so your local builds either use everything directly from cache or instruct a remote builder to produce the artifact for you.

No matter what you change, because the dependencies are graphed precisely, you only need to rebuild a very tiny set of artifacts impacted by your change.

Of course, this doesn't actually work in practice.

Your builds probably aren't deterministic, so your graph of artifacts won't be either, causing lots of stuff to rebuild. Also, it may work great for something like Java that produces class files, but not provide any caching at all for ruby.

Debugging and supporting it is a full time job for a team of engineers that dramatically outweighs the cost of keeping your projects sensibly sized.

You might think that as the project gets larger, it's totally worth it to use bazel for that sweet caching. But in reality the graph construction and querying will become so bloated that just figuring out which targets need to rebuilt becomes a full time engineering effort that breaks constantly with all tooling upgrades.

Also, the plugin ecosystem is just poor.

Bazel is the perfect storm of computer scientists loving big graphs, Google exporting an open source project and then rebuilding it internally, and inexperienced engineers being sold on a tech as being obviously right because all the big players use it.

Re:When to use Bazel? by bobsomers on Tuesday September 13, 2022 @08:32PM (#32831871)
This completely mirrors my experience with Bazel as well. As long as, like Google, you have plenty of engineering effort to waste on nothing but maintaining your build system, sure why not.

But if you actually have to get something done for your business to exist, it's a lot of unrelated work to keep the beast fed and happy.

Re:When to use Bazel? by dietrichepp on Tuesday September 13, 2022 @06:23PM (#32830941)
> Makefile can do all that too,

I guess it's theoretically possible, but why would you do this? This sounds like some kind of "I want to prove it can be done" project like running a web server on a Commodore 64, or making a kitchen knife out of cardboard. Yes, you could figure out a way to download and install dependencies using Make and Curl, and maybe you could build the same library for multiple targets using a bunch of variable expansions, possibly by invoking make multiple times, or including the same makefile multiple times. Make sucks; it sucks a lot; I'd be miserable.

And then if you aren't very careful, you'll end up forgetting to declare some dependency, or some flag change will invalidate your build but Make won't do anything, or you'll make a change to your makefile and forget to clean, and then you spend another hour or day debugging some build that was made out of stale parts. I've done this before, which is why I avoid make for everything but the smallest projects.

> without java dependency

Bazel does not have a Java dependency. I think you might be assuming that because Bazel is written in Java, somehow that means that Java must be installed on your computer. This is not true. Bazel is a self-contained executable you can drop in /usr/local/bin.

I really don't understand your perspective here at all. I've seen people defend make, but make is full of so many traps and gotchas--I wonder how someone could use make and then decide that these traps and gotchas are okay. The projects that successfully use it tend to be small projects, use generated makefiles (automake, cmake, etc), or tend to be a bit simpler.

I use make myself, but as a rule of thumb, only for projects with a few files.

Re:When to use Bazel? by ajconway on Tuesday September 13, 2022 @04:42PM (#32829932)
Bazel doesn't have any user-facing java dependency.

> what real extra benefit Bazel brings in here

Builds are hermetic by default, so unless the developer chooses to escape the sandbox, everything is guaranteed to build on other machines with no additional setup.

(Also, I genuinely hate when I have to manually install build dependencies system-wide and pray that there will not be any conflicts. Having everything pinned to specific sha256 or git hashes by design is a breath of fresh air)

Re:When to use Bazel? by blueblob on Tuesday September 13, 2022 @04:58PM (#32830098)
I used a M1 Mac to try to build tensorflow and tensorflow-text and it is very untrue that everything is guaranteed to build on other machines with no additional setup.

The parenthesized comment is funnier to me because I had to download a specific bazel version to build.

Re:When to use Bazel? by ajconway on Tuesday September 13, 2022 @06:05PM (#32830779)
That is true, Bazel itself is still evolving, and there have been breaking changes between versions. Sometimes the required version number is placed in a .bazelversion file, which makes Bazelisk your top-level dependency.

I'd expect Tensorflow to have some non-hermetic build actions, but if choosing a specific Bazel version was the only thing that was required to build it, that's awesome!

Re:When to use Bazel? by blueblob on Wednesday September 14, 2022 @08:53AM (#32836465)
It was definitely not the only thing that was required to build, but that may be as much Apple's fault as bazel's.
Re:When to use Bazel? by jvolkman on Tuesday September 13, 2022 @05:10PM (#32830220)
> The parenthesized comment is funnier to me because I had to download a specific bazel version to build.

If you use bazelisk to provide your `bazel` command, it'll download the appropriate Bazel version for the repo you're trying to build.

https://github.com/bazelbuild/bazelisk

Re:When to use Bazel? by dmitriid on Wednesday September 14, 2022 @01:34AM (#32833648)
So, more tools on top of your tools. And this couldn't be a part of bazel proper for the dame reason MS built a separate tool to discover paths for their tools: overengineering
Re:When to use Bazel? by jpollock on Tuesday September 13, 2022 @04:45PM (#32829976)
Bazel is more strict about what constitutes a dependency change. It doesn't use mtime on the file, it relies on the checksum. It also considers the command line flags to be part of the cache key.

So, spurious changes (touching a file) will result in a cache hit, while hidden changes (changing an environment flag used by Make) are caught.

This is particularly important if verifiable builds are needed for SoX compliance.

Re:When to use Bazel? by aaaaaaaaaaab on Tuesday September 13, 2022 @04:51PM (#32830028)
What if I use the __DATE__ macro?
Re:When to use Bazel? by jvolkman on Tuesday September 13, 2022 @05:13PM (#32830245)
Re:When to use Bazel? by jpollock on Tuesday September 13, 2022 @05:14PM (#32830257)
Yes, you can go out of your way to make your builds non-deterministic. That should break in Make as well.

I don't know enough about bazel to answer your question definitively, but "stamping" is what you want to search for.

https://bazel.build/docs/user-manual

Re:When to use Bazel? by kmarc on Tuesday September 13, 2022 @05:17PM (#32830288)
The better question is, what if you change a comment?

It checksums the preprocessed files, so neither `__DATE__`, no comment changes should affect your build times.

Re:When to use Bazel? by jpollock on Tuesday September 13, 2022 @05:46PM (#32830556)
I think __DATE__ without redaction pushes the dependency change 1 level down, to the .o file.

The .a/.so/.exe would no longer match the inputs (the .o files), causing a re-link.

Re:When to use Bazel? by vbezhenar on Tuesday September 13, 2022 @06:26PM (#32830966)
Does it re-read every single source file to find out which were changed? Sounds like a terrible approach. Or it uses mtime first and checksum after that?
Re:When to use Bazel? by joshuamorton on Tuesday September 13, 2022 @07:06PM (#32831286)
Mtime first is unreliable on modern (fast) machines, you can edit a file twice in a single second. I think bazel has a daemon watch for file changes, but I'm not certain of that.
Re:When to use Bazel? by simoncion on Wednesday September 14, 2022 @12:37AM (#32833372)
ext4, btrfs, XFS all use nanosecond time resolution.
Re:When to use Bazel? by jpollock on Wednesday September 14, 2022 @12:39PM (#32839666)
Think remote file system where a different node is providing the mtime, so it can have substantial diffs between nodes.
Re:When to use Bazel? by oftenwrong on Wednesday September 14, 2022 @05:20AM (#32834868)
Bazel has a stateful server component that caches information about the build graph and source files in memory. As you said, this allows it to check mtimes first to avoid re-computing hashes for source files.

Bazel can also read source file hashes from the filesystem if the filesystem supports it.

Re:When to use Bazel? by jxramos on Wednesday September 14, 2022 @01:41AM (#32833677)
My understanding is that somehow the bazel server tracks all that and keeps a hot process for monitoring the source files somehow to make the file io more bearable.
Re:When to use Bazel? by kldx on Tuesday September 13, 2022 @04:45PM (#32829966)
Can make do incremental builds based on checksums instead of modification time? How about distributing the build across a cluster in a language agnostic way?
Re:When to use Bazel? by IshKebab on Tuesday September 13, 2022 @05:06PM (#32830188)
The benefits that it brings is the whole reason it exists: precise dependency tracking.

That means you can't have missing dependency links (very common and hard to debug in Make or CMake for example). You can know what outputs a change affects so you can only build/test a subset of the project in CI. Incremental builds become totally reliable. Etc.

Re:When to use Bazel? by jen20 on Tuesday September 13, 2022 @08:24PM (#32831815)
> straightforward to read at any time later

Any makefile doing anything remotely complex is definitely not readable later.

Re:When to use Bazel? by baby on Wednesday September 14, 2022 @04:14AM (#32834493)
Honestly Makefiles that are more than 10 lines should not exist
Re:When to use Bazel? by robbintt on Tuesday September 13, 2022 @08:53PM (#32831997)
Do you have anything OSS that I could try to graft?
Re:When to use Bazel? by dietrichepp on Wednesday September 14, 2022 @01:04AM (#32833504)
I made a demo project that you can build with Bazel. I've tested it with Bazel 5. It should build on macOS and Linux with just devtools + Bazel 5.

https://github.com/depp/bazel-gba-example

Cross-compiling toolchains are a pain to set up in Bazel. Once you get them working, it's nice.