Buck – A build system developed and used by Facebook

Re:Buck – A build system developed and used by Facebook by ProblemFactory on Tuesday March 14, 2017 @05:13AM (#13866081)

I have briefly used https://github.com/Factual/drake for this.

It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover:

* Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways,

* Outputs that are directories with many files,

* Built-in input/output file sharding (for example by date or month) and HDFS support,

* Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc,

* Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by makapuf on Tuesday March 14, 2017 @06:08AM (#13866280)

I'm not commenting about Drake or its usefulness, but concerning (GNU) make :

* you can specify several inputs & outputs per rule

* you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell...

* downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided

* fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps)

and

* "everyone" "hates" gnu make, which means everyone knows it.

(edit :format)

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by Arkanosis on Tuesday March 14, 2017 @06:40AM (#13866397)

AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by lgas on Tuesday March 14, 2017 @06:51AM (#13866434)

Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multi...

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by Arkanosis on Tuesday March 14, 2017 @07:27AM (#13866568)

That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by lomnakkus on Wednesday March 15, 2017 @08:57AM (#13875738)

That doesn't really look like "handling it" by any sensible definition. That looks like a huge kludgy workaround, almost like user implementing a build system inside the build system. (I mean, wtf, asking the user to implement a lock-and-set operation inside their build?)

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by qznc on Tuesday March 14, 2017 @07:29AM (#13866575)

GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by Arkanosis on Tuesday March 14, 2017 @07:32AM (#13866589)

It even has builtin implicit rules. Enabled by default. Checking for the existence of files that are not found in any new project for maybe ten or twenty years (such as RCS and SCSS version control files).

On. Every. Single. Run.

Yup.

[ Parent | on HN ]

Re:Buck – A build system developed and used by Facebook by xyzzy_plugh on Tuesday March 14, 2017 @11:49AM (#13868503)

But I love the built-in implicit rules. I use these frequently.

Often, I write a little test program in C or C++. I name the file "foo.c". Then, to build it I simply type `make foo`.

Sure the rules may be a bit stale, but there isn't much software i use today that I could use in a near identical manner 40 years ago. Make has truly stood the test of time.

[ Parent | on HN ]