We didn't have Bazel on ARM for NixOS because we can't bootstrap openJDK on ARM. There's no source code for a JVM that compiles on ARM afaik. Though in theory we could package binary blobs from oracle and bootstrap JDK and Bazel from there, it means now our trust path for your critical build tool has a random Oracle blob in its trust path there that is extremely hard to get rid off.

Build systems should be easy to build. Bazel is not by a far shot due to java being a mess in this area (ironic given it being advertised as a cross platform language).

These issues are not important for everyone perhaps. (Heck the entire Kubernetes ecosystem pulls random binaries from docker hub without any cryptographic signatures.) but it's still a shame; especially for a build system.

Google builds stuff for themselves and then dumps it for extra reputation points on the dev community.

The part where this starts to suck is when their tools become pseudo-standards and everyone has to live with their decisions.

I'm biased because I work at Google on an open source project, but the complaint here seems to be:

1. A corporation spends significant resources developing a project.

2. Then they release to allow the public to use it at zero cost. Often they even invest in supporting that external use.

3. The project provides enough value to so many users (who chose it of their own volition) that it becomes nearly a standard.

And this is somehow a bad thing?

The complaint is that the tech mega-corps are too big and should be broken up. Then they wouldn't distort markets (among other things) by dumping free products subsidized through ads on them, only to forget about said products two years later.

Regarding your specific interpretation:

1. The product was developed for internal use AFAIK, then released externally.

2. Google is getting something out of this: a PR boost, better engineer retention and the engineers feel good about themselves.

3. All these projects not only don't become standards, but they just add to the pile of things one has to learn because there are still too many people that are fawning over anything published by FAANG.

I think it's unfair to sketch it as such. Isn't it clear that the parent is unhappy with the side-effects, rather than the project itself?

This is actually called the "throw over the fence" open source development model.

Most Google projects are like this, including Android which is the main reason why vendors struggle with os updates.

AFAICT Android is a pain to upgrade because of custom closed-source drivers and lack of ABI stability in Linux, so you need to recompile a driver for a newer kernel.

Android is not a Google's internal product, unlike Bazel.

Last week we had a major security vulnerability in Android just because Google can't user the same Linux kernel that everyone else are using.

There exist LTS kernel versions for a reason, you know.

This. Treble is suppose be their ABI

Right. Like the v8 javascript engine. If you want to build it, you need to download gigabytes of sources and install Googles internal build system tools first...

yes, and often lots of prebuilt build tools no one really knows how to build from source. It's a real mess open source.

They even maintain internal tools that are similar versions, but better. Google uses Borg internally, but they open sourced the cheap knock-off known as Kubernetes instead.

You could try cross-compiling OpenJDK from x86_64 to ARM. This is a lot easier in newer OpenJDK releases than it used to be.

Currently doing this for BSD using OpenJDK 11.

If needed, run with qemu userspace emulation.

> we can't bootstrap openJDK on ARM

Irrespective of Bazel, that's an lot of software than can't run on NixOS ARM, including build-related stuff:

* Elasticsearch, Cassandra

* Eclipse, IntelliJ, Netbeans

* Jenkins

* Java toolchain, Android toolchain

> (ironic given it being advertised as a cross platform language

Indeed that is one of Java's hallmark features. Servers, desktops, phones, toasters.

---

FWIW it's not unheard of to require compiling on one platform for another. This is how most of the embedded world works.

Is not the openjdk available for arm? What do you need from Oracle? https://adoptopenjdk.net/releases.html#aarch64_linux

This is aarch64 (that is 64-bit ARM), and NixOS seems more 32-bit arm looking at their wiki https://nixos.wiki/wiki/NixOS_on_ARM On the other hand, they do 64-bit too, so maybe there would be some way to do for one, then bootstrap the other?

Other idea, distributions like Debian, seems to build OpenJDK as well, so maybe that would be a way to get the initial package (through an existing distro) https://buildd.debian.org/status/logs.php?pkg=openjdk-11 - see the armel/armhf/arm64 architectures (arm64 equals aarch64).

We only support aarch64 officially. 32 bit ARM is only community-supported.

Aren't there 32-bit ARM binaries available here? https://bell-sw.com or here? https://adoptopenjdk.net/releases.html?variant=openjdk11&jvm...

Isn't there a way to get the Android toolchain to give you what you need? After all Dalvik is just Java warmed over and has a working VM, and there are plenty of ARM based Android phones.

That might get you bootstrapped to the point where you can use that to re-compile the community edition of Java.

You are conflating the runtime environment with the development framework. Afaik, you can't develop native Android apps on Android.

This was true for a long time, but we do actually have a bootstrapped openjdk for arm since a few months: https://github.com/NixOS/nixpkgs/pull/65247

You want to build java from source on arm ? Can’t you borrow what AdoptopenJdk Does

Can't you bootstrap older openJDK with (now discontinued) GCJ?

Parts of Bazel are written in C/C++.

The command line client as well as some low level file utilities.

But Bazel is distributed as a standalone package in several packaging systems; no need to install Java yourself.

Though I doubt many people know this; there's virtually no reason you would be exposed to its language implementation choices.

EDIT: Actually I believe Bazel is changing its .deb package to not be standalone, in order to comply with Debian requirements.

> there's virtually no reason you would be exposed to its language implementation choices

Um, doesn't it still have a client that starts up a persistent server and you then have to wait for that server to start up (which, being Java, takes forever) and then deal with those server processes hanging around?

That sounds like being exposed to its language implementation choices.

As opposed to, say, having fast start-up times and not using a server design like basically every other build too.

But my information might be out of date. That's what always soured me on it previously, though.

Bazel doesn't start a server for Java JIT reasons.

It starts a server for (1) concurrency control (2) management of worker processes some languages use (3) caching the build graph (recall that Bazel works with very large code bases).

These reasons are independent of implementation in C++, Go, Rust, Java AOT, etc.

(And yes it doesn't have to use a persistent process to solve these problems. That is the solution it chooses.)

> not using a server design like basically every other build too

Buck, Pants, sbt, Gradle

> (And yes it doesn't have to use a persistent process to solve these problems. That is the solution it chooses.)

How much did the fact that Java is very slow starting up influence the decision to use a persistent process instead of some other solution?

Java starts up in ~1s, something like Graal could make this faster. But I find this Java criticism more a symptom of Java-derangement syndrome, because python and node build systems have even worse startup times and no one says anything.

> python and node build systems have even worse startup times and no one says anything.

Noop build with Waf (build system written in Python) takes 0.13s on my system. Waf reports that the build actually takes 0.04s, so I guess 0.09s is Python's start-up time (and some other overhead).

Java Hello World takes 0.132s on my Macbook pro. If I turn on -Xshare:on to use class data sharing, then it drops to 0.119s. Ergo, Java startup time is non-factor. Graal could make this even quicker, for example, a GraalVM AOT helloworld can be reduced to .008s startup, see https://github.com/graalvm/graalvm-demos/tree/master/java-ko... for example.

"time bazel" returns 0.098s

Running a null build took 0.84s, but Bazel does significantly more work, as it's working on a big monorepo, as both a build tool and a package manager.

In short, it is not a problem. But I've had npm take many many seconds for simple operations. "npm list" takes 2.9 seconds.

Python can be slow, depending on the tool. Waf sounds like it is simple and fast, but there are lots of other examples of slow python frameworks out there.

> "time bazel" returns 0.098s

For me it returns 1.6s on first run, .9s on second (bazel 0.29.1).

    $ time (echo 'quit()' | python)

    real    0m0.025s
    user    0m0.000s
    sys     0m0.031s

    $ time (echo 'quit()' | python3)

    real    0m0.036s
    user    0m0.031s
    sys     0m0.016s

(both are after it was already cached)

Huh; I see < 0.15s.

> so I guess 0.09s is Python's start-up time (and some other overhead)

That is probably mostly Waf startup time. Python itself starts way faster than that (on the order of 10-20ms on my machine).

Startup time in any interpreted language is very quickly simply proportional to the amount of code loaded.

   time python -c ''                               # 0.024s
   time python -c 'import argparse'                # 0.036s
   time python -c 'import argparse, json'          # 0.040s
   time python -c 'import argparse, json, httplib' # 0.065s

Indeed, but it isn't a cool toy among this crowd.

My guess is that startup time is not the issue, but loading a large data structure (its cache) from disk can be.

Especially if you can't or don't want to use memory mapping because it's hard to do well.

But more importantly, I think that the server can monitor file changes ahead of time with things like inotify, saving the time of stat()-ing files when the user wants to perform a build.

Change monitoring with inotify is going to hit limits quickly.

I recently wanted to take a look at VSCode and it immediately shat itself over the maximum number of inotify watches being too low (the kernel I'm running restricts this to 8k for non-root users).

I then bumped the limit to the max (512k, I think, or about 7 copies of linux.git)…still no luck. Now imagine you have a Google-scale number of files.

> Especially if you can't or don't want to use memory mapping because it's hard to do well.

Why do new languages never address this issue?

In-kernal block caching of files is pretty crude - since the kernel has no knowledge of your data structures or which bits of your files you're going to be reading next, you end up having hundreds of page faults requiring single-sector disk reads for most workloads.

That’s often the case if your file format evolves without taking a mmap use case into account. But for a format that’s designed with mmapping in mind, it is often ridiculously faster and more effective.

But most languages and environments don’t let you do that easily.

Aren't the hot compilers kept around under this mechanism to speed up the next incremental build round? I think it's not just caching of intermediate build output but also not letting the tools start from a cold start.

Yes, and I included that in my point. Certain compilers: Java, Swift, TypeScript are faster with long-running worker processes.

To be fair, that doesn't require a long-running Bazel process to manage them, but that does become a natural choice.

IDK.

I do know that Go wasn't an option :)

(Because it didn't exist)

JVM startup is sub-second. Still not great for e.g. small command line utilities, but completely fine otherwise.

JVM startup for `java -version` is around 150ms on my machine:

    $ time java -version
    openjdk version "11.0.1" 2018-10-16
    OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
    OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

    real 0m0.159s
    user 0m0.116s
    sys 0m0.052s

This is how long it takes for a program like git to do actual work. Once you start running an actual program like Maven, it blows up to 500ms for `mvn -version`:

    $ time  mvn -version > /dev/null

    real 0m0.570s
    user 0m0.431s
    sys 0m0.145s

Bazel takes 150ms to do nothing:

    $ time bazel --version > /dev/null

    real 0m0.144s
    user 0m0.028s
    sys 0m0.063s

As a baseline, make takes under 30ms to do nothing:

    $ time make -version > /dev/null

    real 0m0.025s
    user 0m0.002s
    sys 0m0.016s

Seems similar on my system. Some more comparisions:

Waf written in Python:

    $ time ./waf --version >/dev/null

    real 0m0,068s
    user 0m0,056s
    sys 0m0,012s

Ninja written in C++:

    $ time ninja --version >/dev/null

    real 0m0,001s
    user 0m0,001s
    sys 0m0,000s

Java 13 takes 97ms on Windows 10, and I haven't bothered to produce a Java native image before attempting it.

    PS C:\Workdir> Measure-Command { java -version }
    openjdk version "13" 2019-09-17
    OpenJDK Runtime Environment (build 13+33)
    OpenJDK 64-Bit Server VM (build 13+33, mixed mode, sharing)


    Days              : 0
    Hours             : 0
    Minutes           : 0
    Seconds           : 0
    Milliseconds      : 97
    Ticks             : 975572
    TotalDays         : 1.12913425925926E-06
    TotalHours        : 2.70992222222222E-05
    TotalMinutes      : 0.00162595333333333
    TotalSeconds      : 0.0975572
    TotalMilliseconds : 97.5572

About 60 ms more than make.

Hurray Make still wins, now what are we going to do with those 60ms... /s

FWIW, I don't think "java -version" is a good benchmark of java startup time - why should it start up the JVM just to print the version?

I think that measuring anything less than 1s for CLI is silly, school playground measuring competition.

The original comment was "JVM startup is sub-second. Still not great for e.g. small command line utilities, but completely fine otherwise."

For tooling it matters. Some scripts will have backticks and $() and each use of these could be up to a second and be acceptable? Not really. And if you want to run a command like `go fmt` every time you save a file in $EDITOR then you want it to be fast. Maybe when we use the Language Server Protocol everywhere, it will be a different world. But a lot of editors shell out for these features still - and you would never dream of shelling out to e.g. `mvn dependency:tree` because it's too slow.

Unless I am missing something, 97ms looks pretty much sub-second to me.

With Graal AOT you can make command line utilities that start on the order of 0.008s

https://github.com/graalvm/graalvm-demos/tree/master/java-ko...

> make takes under 30ms to do nothing:

But in large code bases, make can take several seconds to do nothing, depending on the number of `$(shell find)`.

Whereas Bazel's time remains constant. Could it have been constantly 30ms instead of 150ms?

Sure. But that's nowhere near to being a problem for the usual code bases Bazel operates on.

$ time ./java -version openjdk version "14-ea" 2020-03-17 OpenJDK Runtime Environment (build 14-ea+19-824) OpenJDK 64-Bit Server VM (build 14-ea+19-824, mixed mode, sharing)

real 0m0.074s user 0m0.063s sys 0m0.025s

Does the initial startup time matter?

I think the server is kept around for a number of reasons that have nothing to do with Java startup time. Probably the biggest one is file-watching, using inotify, FSEvents, etc.

https://stackoverflow.com/questions/57982028/how-does-bazel-...

The server shuts itself down after some idle time, I’ve never had to actively manage any instances of it.

I always have the opposite reaction. When I see its a native tool I immediately think, "oh great, now to build this I need to know how to understand your weird glib / gcc version weird compile / link / error for some ecosystem I don't understand". Instead when I see Java / JVM and know that almost certainly I will have no problems like that.

Eh, static binaries solve this & Gradle presents plenty of potential problems in my experience.

There are alternatives that satisfy this (valid, IMO) requirement. For example, there is build2 (https://build2.org ; full disclousure: I am involved with the project) that tries to achieve the same objectives (correct & fast builds, uniform interface accross platforms, being language agnostic, etc) though not necessarily using the same mechanisms (build2, for example, tries to be fairly light-weight). Here is the introduction if anyone is interested: https://build2.org/build2/doc/build2-build-system-manual.xht...

I checked as I was interested, but it seems that build2 doesn't do distributed builds.

edit: remote->distributed

Not yet, but it's coming. And most of the really difficult infrastructure is already there, at least for the C/C++ compilation.

I think you don't need to install Java to use Bazel, it comes bundled. They have done some improvements to installer so that the user requires as less additional tools as possible to run Bazel.

>I only wish Bazel had been written in a language that produced native binaries like C++ or Go instead of Java.

Java can be compiled into native binaries using GraalVM. Not to mention you dont actually have to install Java as it comes in bundled.

native-image performance without PGO might be worse than with JIT, at least in some experiments that I did.

See also https://github.com/oracle/graal/issues/979

I have the same complaint about meson and python 3. You purposely use a low-level language to write dependency-free code, then your users need to install a bloated runtime just to build the damn thing.

You should look at Please: https://please.build

Bazel-like in many ways; written in Go.

Bazel has been embedding Java for a year or so now. There's no need to concern the host environment with installing the JRE anymore. As a user, the fact that Bazel is written in Java is transparent.

You don't need to install Java. It's bundled. In fact I'd recommend that you don't even install Bazel itself. Instead, install this: https://github.com/bazelbuild/bazelisk. This will install either the latest Bazel for you automatically, or you can pin a version of Bazel you want in each repository, to limit and control build system churn.

It's funny how Java people always marketed Java as "write once, run everywhere" and they often bundle compatible platform-dependent JRE with platform-specific plugins. ^_^

Is that such a big issue though ?

The default install of my dev machine (Ubuntu) installed two different versions of python and perl5, neither of which I use. Other languages have been installed indirectly because I installed one tool or another. As long as the language itself doesn't get in the way (e.g. the tool is picky about the language version and requires extra work to install it) why should I care ?

I bet you're using Python and Perl indirectly, because lots of Ubuntu's tools are written in those languages.

You can use SDKMAN (https://sdkman.io/usage). It is like nvm.

There are native code compilers for Java, one just has to bother to learn about them.

Plus, even if one does not use them, shipping a minimal runtime is hardly any different from having libc or libc++ dynamically linked.

I am going to be trying out Bazel next week and I intend to run it in a container. I assume this is possible?

I guess that's why they use "gn" in newer projects like Fuchsia or Chrome?

Why do you care if a JVM is on your dev machine?

- One more thing to install - Does this work for 8/11/12/13/14/15/16?

It's a stripped down JVM is bundled with bazel, you don't need to install it.

My 2015 MBP is crawling between Safari, Docker, VS Code, and BitDefender (antivirus). Not sure how bulky the JVM is these days. That said, I wish it supported Python.

Why do you even have Bitdefender installed?

A lot of people have it for compliance reasons.

A lot of regulations are actually a lot more reasonable than one might think and refrain from mandating any specific technical measures.

Yet, people always seem to point to regulations to justify all kinds of snake oil salesmen garbage, even if obviously unfit to fulfill the stated objective.

If you are building software for others to consume it is a good idea to have at least some protection. Even it doesn’t ever pickup MacOS malware/virus - it might stop you from distributing that intended to infect other target OS or software.