Buck – A build system developed and used by Facebook

Re:Buck – A build system developed and used by Facebook by m_mueller on Tuesday March 14, 2017 @03:13AM (#13865646)

So far I've also always found `make` to be the best option for what I need, especially because it's already available everywhere (albeit sadly in very subtly incompatible versions).

Anyways, one thing that has always bothered me about make is that it so much depends on file modification dates. Imagine if it would instead use a very optimised hashing algorithm over the input content. Content can be a file or any URI, so it uses wget/curl and ssh as a dependency. Hashing is optimized such that it fails early - e.g. hash in n kB increments, mark as new and return if changed.

Imagine how well this would now suddenly integrate into our modern web service based landscape. You could hook together services quite easily:

    all: report.csv

    report.csv: ${customers_json_uri} ${earnings_json_uri}
        ${report_executable} $^ > $@

Has anyone built something like this already?

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by LegNeato on Tuesday March 14, 2017 @03:51AM (#13865786)

Buck (and Bazel) essentially do what you want and more:

- Don't use modification times and instead do intelligent hashing (see https://buckbuild.com/concept/rule_keys.html)

- Include the compiler and a bunch of other stuff in the hash. This means you can share cached artifacts safely via a http cache (https://buckbuild.com/concept/http_cache_api.html)

- Can refer to remote files (see https://buckbuild.com/rule/remote_file.html)

- Can use any script to do anything as long as it has a single consistent output (see https://buckbuild.com/rule/genrule.html)

Because their model is so much better, you get crazy fast local caching and remote shared caching of artifacts.

Please don't use make, the newer build tools are better in virtually every way.

(disclosure: my team created Buck when I was at FB)

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by ProblemFactory on Tuesday March 14, 2017 @05:13AM (#13866081)

I have briefly used https://github.com/Factual/drake for this.

It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover:

* Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways,

* Outputs that are directories with many files,

* Built-in input/output file sharding (for example by date or month) and HDFS support,

* Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc,

* Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by makapuf on Tuesday March 14, 2017 @06:08AM (#13866280)

I'm not commenting about Drake or its usefulness, but concerning (GNU) make :

* you can specify several inputs & outputs per rule

* you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell...

* downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided

* fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps)

and

* "everyone" "hates" gnu make, which means everyone knows it.

(edit :format)

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by Arkanosis on Tuesday March 14, 2017 @06:40AM (#13866397)

AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by lgas on Tuesday March 14, 2017 @06:51AM (#13866434)

Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multi...

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by Arkanosis on Tuesday March 14, 2017 @07:27AM (#13866568)

That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by lomnakkus on Wednesday March 15, 2017 @08:57AM (#13875738)

That doesn't really look like "handling it" by any sensible definition. That looks like a huge kludgy workaround, almost like user implementing a build system inside the build system. (I mean, wtf, asking the user to implement a lock-and-set operation inside their build?)

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by qznc on Tuesday March 14, 2017 @07:29AM (#13866575)

GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by Arkanosis on Tuesday March 14, 2017 @07:32AM (#13866589)

It even has builtin implicit rules. Enabled by default. Checking for the existence of files that are not found in any new project for maybe ten or twenty years (such as RCS and SCSS version control files).

On. Every. Single. Run.

Yup.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by xyzzy_plugh on Tuesday March 14, 2017 @11:49AM (#13868503)

But I love the built-in implicit rules. I use these frequently.

Often, I write a little test program in C or C++. I name the file "foo.c". Then, to build it I simply type `make foo`.

Sure the rules may be a bit stale, but there isn't much software i use today that I could use in a near identical manner 40 years ago. Make has truly stood the test of time.

[ Parent | on HN ]