Assuming you're building your own code and are willing to arrange things in a compatible manner, then I have a solution to offer. It figures out the dependencies by reading the source code. You can create objects and binaries just by asking it to build a certain target.

I recorded an example of installing and using it here:

http://rachelbythebay.com/jvt/view?bb_install

You can get a copy for experimentation here:

http://rachelbythebay.com/bb/

It isn't for everyone, but only you can know if it'll meet your needs. Hopefully this helps.

Whoa, your terminal playback thing is pretty neat. Did you use GNU Screen to record the session?

Thank you! I used script. Certain implementations have an option to emit timing and byte count data on stderr. Then it's just a matter of saving it and building something to honor those delays during playback.

That's awesome! I had no idea about `script -t`; I'd written my own a few years back. Thank you!

You can rip it off like this: https://github.com/ysangkok/terminal_web_player

Pretty majorly uncool to do that and not give credit or even a link to rachelbythebay.

> willing to arrange things in a compatible manner, then I have a solution to offer.

So you're proposing that people give up screwdrivers because figuring out what bit to use is hard, and that they should instead use your potato peeler, never mind that it doesn't actually drive screws?

Nope. Not at all. But I like that you think that.

That is very cool, and it's something that will only get better if the Apple-proposed modules get widespread.

Thank you! I'm not sure what you mean by "Apple-proposed modules", though. Please let me know more and I'll see what I can do.

Is the source available?

If you mean the source to my replay stuff, well, it seems you have already established a way to snag that. The only original part of that was my wrapper to the terminal to fetch byte streams and replay them while honoring the timing data.

Regarding the build tool source, I haven't decided what to do about that just yet. It could be particularly valuable in a corporate environment.

I debated whether to start namedropping build systems, but I decided it was going to be counter productive and devolve into a debate of the relative merits of the systems I'd picked out. In the end it depends heavily on the environment you're in what makes sense for your project.

In other words: if you're not sure, use make.

No, really not. If you pick a build system at random it's probably better than make. If it's a big project it's worth taking 30 minutes to actually look at what's available and pick one.

I'd suggest tup[1] and Shake[2].

[1]: http://gittup.org/tup/ [2]: http://community.haskell.org/~ndm/shake/

Thanks for the links! I will definitely use tup for some next project. I try from time to time different make-like systems but I always come back to GNU Make. I also don't fear the GNU Make Reference. But tup looks, again, quite promising. It even has those little context sensitive one-special-char one-letter variables :) But they do something better by design, I see http://gittup.org/tup/make_vs_tup.html

> This page compares make to tup. This page is a little biased because tup is so fast. How fast? This one time a beam of light was flying through the vacuum of space at the speed of light and then tup went by and was like "Yo beam of light, you need a lift?" cuz tup was going so fast it thought the beam of light had a flat tire and was stuck. True story. Anyway, feel free to run your own comparisons if you don't believe me and my (true) story.

The completely unprofessional tone here really turns me off to the entire system. If you write like a typical teenager, you probably code like a typical teenager, and I don't want a typical teenager writing my goddamn build system.

Besides: who the hell is bottlenecked on the build system? The compiler and linker (or the equivalent for your favorite language) do all the work. Anyone who believes this article makes a different is completely ignorant of Amdahl's Law.

You might find my paper more informative and less unprofessional: http://gittup.org/tup/build_system_rules_and_algorithms.pdf

Many projects are bottlenecked on the build system. You can benchmark this by timing a null build (running 'time make' after building everything). Some examples from my machine are the Linux kernel (28 seconds), and Firefox (1m 23 seconds). Some of this time is from unnecessarily recompiling things, but that is a separate issue from the inherent lack of scalability in make.

Suppose I want to change a single C/C++ file in one of these projects - the total turnaround time from when I type 'make' to when the build finishes can be described as:

T(total) = T(build system) + T(sub-processes)

Ideally T(total) would be zero, meaning we get an instant response from when we change the file to when we can test the result. Here, T(build system) is the null build time, and T(sub-processes) is the time it takes to run the compiler and such. Using the Linux kernel as an example again, compiling fs/ext3/balloc.c takes 0.478 seconds. In comparison to the null build of 28 seconds, there are significant gains to be had by optimizing T(build system).

Amdahl's Law is a little tricky to apply since tup is not parallelizing T(build system), but rather changing it from a linear-time algorithm to a logarithmic-time algorithm. So you can set P easily based on the relative values of T(build system) and T(sub-processes), but S is not a simple "count-the-cores" metric. The speedup is effectively N/log(N), where N is the number of files. This is much better than simple parallelization - T(build system) for tup with these projects is only about 4ms. The total turnaround time for the balloc.c file in the Linux kernel is 1.1 seconds (which includes compilation and all the linking steps afterward), in comparison to make's total turnaround time of 29.5 seconds.

For very large projects the build system can quite easily become the bottleneck when just changing a single file, which also happens to be the most important use-case for developers. In extreme cases a no-op build with make can easily get to 15+ seconds.

> In extreme cases a no-op build with make can easily get to 15+ seconds.

I have never seen cases so extreme, but my opinion on the matter is that this is a "build smell". If the Makefile has to resolve a DAG this large, that means that developers have to worry about compile- or link-time interactions this large, as well. 100k source files all linked into a single executable is more complex than 10k source files split across 10 executables, and a handful (say <100) of headers which represent "public" APIs. Because if you have 100k source files and your developers haven't all killed themselves already, then there are some firewalls separating various modules already. Formalize it at an API level and split apart the builds, so that it's _impossible_ for anything outside of the API itself to trigger a full rebuild.

Typically this shows up in recursive make projects with lots of sub projects—it doesn't take that much time to stat every file in question but reinvoking make sixteen times can be quite slow.

I don't deal with this by not using make, I deal with this by not writing recursive makefiles.

> it doesn't take that much time to stat every file in question but reinvoking make sixteen times can be quite slow.

Yes, reinvoking Make repeatedly tends to force redundant `stat` calls. But I have worked in environments where heavily-templated code was hosted over a remote filesystem, and every `stat` call was something like 10msec. That adds up _extremely_ fast, even with non-recursive make. Ugh.

> In extreme cases a no-op build with make can easily get to 15+ seconds.

Most developers will never see such a system. Optimizing for that kind of scale at an early stage has all the problems of any other premature optimization. It's most important to just get the build system out of the way so you can get your real work done, and you do that by writing makefiles, since makefiles are universally understood.

Now, when a project does grow to the proportions you mention, you can start looking at alternatives --- but I'd argue that these alternatives should amount to more efficient ways to load and evaluate existing makefile rules, not entirely different build paradigms. Make's simplicity is too important to give up.

You dislike his humor, that's fine. Calling it unprofessional is subjective. Some professional environments with great professional output appreciate humor.

Also, I am bottlenecked on my build system at my workplace, which takes ~45 seconds to realize nothing needs to be done (It isn't "make", because "make" does not support our build process).

I do enjoy reading through tup's site

I like how he uses Sauron's All Seeing Eye as a drop in replacement for gods algorithm.

also excellently documented... not like shake, at first

Scons: http://scons.org/

CMake

Personally I'd suggest Premake because it uses Lua instead of rolling its own scripting language. The world would be a better place if we could all agree on one scripting language for stuff like this so no one has to look up the syntax for things like creating arrays for every individual tool.

Feature-wise premake isn't entirely caught up to CMake but it has everything important. Also the premake files look a lot cleaner and more readable compared to cmake files.

> The world would be a better place if we could all agree on one scripting language for stuff like this

It's called bloody "make". If Prolog is a language, so is make. Make is still a scripting language even if it's not boneheadedly sequential and imperative the way, say, Python is.

Make is simply not up to the job. It make writing correct build systems very difficult (hard to not under-specify dependencies). It does not support auto-generating code and then scanning it for extra build dependencies. It is a crappy tool and we should standardize on something better.

If what you want is a standardized language Scons has that and is a bit more established.

CMake is pretty ugly in its own right, though I will grant it's hard to be worse than GNU make.

CMake is an abomination

Mind pointing out any constructive arguments against it? I recently switched over from hand-built makefiles to cmake for one of my projects, and it's been a breeze.

I prefer autotools to CMake. Maybe I didn't really give CMake a fair shot, but in a few hours of trying, I couldn't out figure out the CMake equivalent to a bit of custom glue code in configure.ac that checked for a Lisp compiler.

I get the feeling CMake works fine as long as you color within the lines, but that it's much harder to extend than autotools is.

Yes, this is something I have thought through, though I may have been lucky/unlucky with my pick of projects. But generally speaking, installs I have installed based on GNU autotools, seems to install with less fuzz than those with C-Make. So, that, and the fact that there is full free documentation of GNU Autotools, even books out there, have made my personal choice easy.

For me the overall factor is the ease for any of my users, installing something I have written.

Using a macro language was already bad in the 90ies (autoconf/automake using m4), but using one in 2000 is just tragic.

It is also impossible to debug, partly because of the abomination the cmake language is: even understanding where a variable is defined is hard, and because the language is so unexpressive, the Find*.cmake modules are often in the 1000s lines count.

For all its suckiness, I take autoconf/automake over cmake.

I'm sorry, but your comment does not make much sense to me. autotools is mostly written in m4, with some shell snippets. Those are macro languages. If you don't like macro languages, logically you should not like autotools.

CMake is not "impossible to debug." In fact, it is a lot easier to debug than autotools-- partly because you end up writing so much less code. Also you don't have the three levels of "generated files generating other file generators" that you do in autotools.

For all its suckiness, I take autoconf/automake over cmake.

Well, we agree on one thing. autotools does suck.

My point was that autotools had the excuse of being written in the early 90ies, cmake doesn't.

Your experience with debugging autotools vs cmake does not match mine: cmake is not an improvement over autotools (if only because at least with autotools, there is some decent doc out there and google knows a lot about autoconf insanity). It took me hours to debug trivial issues with cmake, because you can't easily trace where variables are defined.

I have never been able to figure out how to cross-compile a project that uses CMake. GNU Make lets you just set some environment flags and drop in a different compiler.

CMake honors the same environment variables as Make. Just set them before running CMake.

CMake also supports cross-compiling. www.vtk.org/Wiki/CMake_Cross_Compiling

Ninja.

You probably don't want to write ninja files yourself. CMake + Ninja is a nice combo.