For the local edit-test cycle incremental builds are great, but any decent IDE will do them in the multirepo case just as in the monorepo case.
Incremental builds are great for quick day-to-day development, but in my experience they are invariably unsuitable for robust production builds. For me, those should use a git clone or equivalent into an empty directory and then a completely clean build of everything, always.
I wouldn't care if it was reliable. My experience is that it isn't.
> If you don't trust bazel, people have been using distcc+ccache since forever.
And it's been causing inconsistent builds all the while.
I think you're falling into the trap of "past solutions didn't work, therefore the problem is intractable", which is a special case of the "throwing the baby out with the bathwater" fallacy. Sometimes a new tool comes out that fixes the problems of past tools.
Where's the breakthrough then? What's the magic that allows bazel to succeed where dozens of similar efforts have failed? Sometimes a new tool does fix the problems of past tools, but that's almost always due to some new insight.
* Bazel's FAQ: https://bazel.build/faq.html
* A description on how Blaze works: https://google-engtools.blogspot.com/2011/08/build-in-cloud-...
Unfortunately the design doc for Blaze is internal and AFAIK not available publicly. I imagine they took a look at a number of the similar efforts and made decisions based on what those approaches did wrong.
Doing a clean build for the final artifact that is going to be released is pretty standard practice, IME.
If you think about it, a package manager workflow (like maven, nuget, etc) is also a form of incremental compilation. You compile your libraries only once and publish them in binary form in some sort of repository and then import them. The difference is that if you want to make a change in both a library and an application this becomes hard to do with packages, while very easy with a monorepo and Bazel.
The difference is that that's explicit and visible, rather than being "magically" figured out by the build system. And the compiled binaries are 100% reproducible everywhere since it's literally the same file from the repository everywhere. So it's much more reliable, understandable, and debuggable.
> The difference is that if you want to make a change in both a library and an application this becomes hard to do with packages, while very easy with a monorepo and Bazel.
Having a library and application so tightly coupled that changes have to be simultaneous is poor practice, and will likely get you into trouble when the time comes to deploy them. Better to enforce that you test old application + new library (or vice versa) up front, than hit it for the first time in production.
Not at all true in my experience. I use Buck, which is very similar. It constantly recompiles stuff it doesn't need to by any standard. I've seriously considered avoiding Mercurial bookmarks in favor of manual patch management to see if I can avoid triggering the cases where it gets stupid. Maybe it's a problem specific to Buck, but my conversations with peers at other companies suggest not (and only a tiny percentage of developers at any of these companies get a choice of which tool to use).
The other issue is that "more than needed" is subject to more than one interpretation. There's "more than needed" as code was actually written, and there's "more than needed" as code should have been written. Tooling can help with the first but not the second. If developers are sloppy about how their header files are structured, then a change to one that's widely used will make it "necessary" to recompile a lot of code - including code that didn't actually use the part that changed. Oddly enough, though, that "necessary" recompilation wouldn't occur if those same header files were split up. (BTW some languages make good design more difficult by requiring private implementation details in public header files, but that's a separate conversation.) Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.
To find out how/where the dependency was introduced, you can use bazel query 'somepath(your_target, some_other_file)'. Not sure if Buck have an equivalent, but bazel should work with Buck repo.
Sometimes it's unintentional, for example an include left in after it's no longer needed. Other times it's intentional but lazy. For example, I often see commits that pull in one dependency just to use one tiny little utility function, but that dependency pulls in ten others, which each pull in ten others, and before you know it any change in a hundred files forces yours to be recompiled even when that tiny utility function wasn't touched.
A responsible developer would refactor the header file to keep the number of spurious rebuilds down, but the world is full of irresponsible developers. With multiple repos, the public immutable part of an API and the private fluid part tend to find their way into separate files pretty quickly. With a monorepo they don't, which is kind of ironic because one of the purported benefits of a monorepo is that it's easy to reach into "someone else's code" and refactor it. Unfortunately, people just don't seem to use that ability for good often enough unless review culture forces them to.
In practice this is almost never because of "a monorepo" and more because people don't like being good about including what they use (see: any Titus Winters talk ever). If, for example you have :test_foo and :foo_lib, things are fine and dandy. When you modify foo_lib you'll run :test_foo and nothing else. But often people are lazy and you have :foobarbaz_lib which is built from three source files, foo, bar, and baz. Each of those has a test, but all the tests depend on :foobarbaz_lib, because that's all that exists. So when you change foo, you run :baz_test, even though in practice, baz might depend on foo, and not the reverse. You fix this by making :foo_lib, :bar_lib, and :baz_lib each with their associated :target_test and explicit dependencies, and then :foobarbaz_lib depends on all three libs (or doesn't exist at all, and everyone who needs :foo_lib uses :foo_lib directly.
So yes, absolutely, when/if you start (essentially) globbing your imports, bazel/buck/pants stop being awesome. So don't do that.
I think you mean in the practice you've seen. I've seen some practice too, and it seems much more prevalent in monorepos. Yeah, still a small sample, correlation does not equal causation, yadda yadda, but I still think it's because monorepos make it so damn easy to reach out and add a huge tree of dependencies without having to think about it. My whole point is that it's possible to do the right thing, but requires more discipline.
Keep in mind that the alternative is "I have to build (or pull a prebuilt version of) the entirety of every repo that I depend on". An entire repo is always going to be larger than a single target. Yes you're right that pulling in the entirely of {{some large tool}} because you want `tool.strings.util.mangle_name` is bad.
The alternatives are
1. (multi-repo) You stick mangle name in its own repo (think `left-pad`)
2. (monorepo) Depend on `tool` in its entirety
3. (either, though more common in multi-repo) Everyone re-implements mangle_name. If the way the name is mangled is important, and it changes in the future, you have major issues and incompatibilities. (think IPV4 -> IPV6 for an example where a "global" `parse_addr` function would have made everyone's lives easier). You can bet on the assumption that mangle_name will never change, but then you aren't doing software engineering[1]
All of these are the wrong solution
4. Core utilities like this get factored out into their own files. In a multi-repo environment, everyone depends on the entire core/util repo. In a monorepo environment, you depend on the granular files you need.
4 is the only correct solution, and note that in a monorepo environment it results in you depending on fewer things than in a multi-repo environment. Yes there are a lot of ways to do things wrong, But that's true no matter what tooling you use. When used "correctly", monorepos have the best outcome, and the bad outcomes aren't actually that bad (sometimes longer builds and binary bloat if you don't tree-shake). You always need discipline. Monorepos don't need more.
A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. This is extremely efficient. In fact, in my company with a few hundred projects we tried it and it was 10 times faster than our previous Maven based workflow. These tools also help you with dependency management so you keep your code organized. The killer feature is that you finally have true Continuous Integration because the build tool will figure out any dependencies of the code that you change, rebuilding it and testing to make sure that nothing is broken.