News for hackers. Stuff that matters.
Re:Ask HN: In what creative ways are you using Makefiles? by gopalv on Monday July 24, 2017 @03:37PM (#14841907)
To generate 100 Terabytes of data in parallel ... on Hadoop

https://github.com/hortonworks/hive-testbench/blob/hive14/tp...

The shell script generates a Makefile and the Makefile runs the hadoop commands, so that the parallel dep handling is entirely handed off to Make.

This make it super easy to run 2 parallel workloads at all times - unlike xargs -P 2, this is much more friendly towards complex before/after deps and failure handling.

There are no comments on this comment yet.