See here: http://gittup.org/tup/ex_a_first_tupfile.html
> The trick is that tup instruments all commands that it executes in order to determine what files were actually read from (the inputs) and written to (the outputs). When the C preprocessor opens the header file, tup will notice that and automatically add the dependency. In fact, we don't have to specify the C input file either, but you can leave that in there for now since we'll use it in the next section.
Writing a tup build script doesn't really feel like just writing a shell script that does the thing you want. It's very cool technology and fun to play with, but I didn't find it fundamentally easier to use than other build systems.
To be honest, the only time I haven't felt like an army of build rules isn't a weight I have to carry uphill is when I use Lein (which is more of a build project tool as opposed to something as elemental as tup).
Systems like SBT and Ant end up with things that feel to me like a complete and total reliance on hand-tooled plugins as opposed to a reliance on executing pre-existing executables. Gulp is the same way; even though there are perfectly serviceable and stream-like executables on every OS ready for use, Gulp insists you write more javascript to glue things into its awkward model.
Maybe that's not the end of the world (as opposed to yet another make-like DSL), but in cases with truly and almost comically incomprehensible codebases like SBT (oh SBT team, every time you "simplify" your codebase you totally miss the point we don't care about your abstractions and just need to write command extensions) or Gulp (oh Gulp team, please make Gulp more like Storm/Trident or Hakyll's buidler arrows so it does more than just compile javascript and sass) then you're totally out of luck.
And when we consider GNU Automake and Autoconf that do have a ton of prebuilt rules, we see an even more dire situation: where the language describing the rules is more complicated than the underlying concepts they service.
The only other thing I like about Gulp is that it works without too much hassle on Windows and *Nix.
Other than that, it's dung. Though it's certainly better than Grunt, it's not objectively good. That seems to be a problem with a lot of things these days, "relative better than the alternatives, not something you'd actually want on its own merits."
I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json).
For the life of us, we could not find a way to make Gulp do this without essentially writing a solid block of Javascript that just does exactly what we wanted (in which case, why use anything but the shell script approach?).
I understand that stream rejoining is a complex problem, but Gulp seems ideologically opposed to a more generic approach here. They get mad when people try to inject non-file based values into the system, and actively close off alternatives and shame them on their github issues.
Gulp is a complete mess, in my opinion.
If we rely are devoted to using reusable and small components then a shell that can directly wrap our standard libraries (and has a type system rich enough to express maps and tree types) is indispensable to truly reusable software.
Then what a make system should be is a clear and smart way to express dependencies and workloads.
A shell's primary job should be to support interactive use, and fast development of one-time-use scripts off the cuff, because those are what most of us do most commonly. PS doesn't support either all that well.
People often offer that "plain text is superior..." but neglect to recognize that PowerShell's model actually offers as superset of the current functionality of Bash, and that when object streams are required they are utterly indispensable.
It's also a uniquely eurocentric conceit that text is a simple stream of bytes. At the very least, it is a stream of bytes with an encoding label. Unicode is everywhere, and for some people is required to express their language. You can no more safely assume ISO-Latin-1 as you can UTF-8 or UTF-16.
Text isn't as simple as it used to be when we didn't give a damn about half the world's population.
> Maps and trees aren't as important as you make them out to be. You can do key:value stores in text at O(n) cost, which is often G ood Enough, and rarely needed.
We live in a world where Curling JSON objects or (even more problematically) protobuffs objects is the de-facto way we interact with the vast majority of remote services. Feel free to try and fit every workflow you have into jq. Feel free to keep working in an environment where even functional return values are not allowed and you keep repeating the same commands over and over.
> And when you really need the extra power, take your pick: Python, Perl, Ruby, SCSH, and countless others.
Quick question though: how is it this any different from the plugin approach from gulp? :)
> A shell's primary job should be to support interactive use, and fast development of one-time-use scripts off the cuff, because those are what most of us do most commonly. PS doesn't support either all that well.
Besides the awkward subquotation escapes for passing to commands, what exactly are you saying it can't do? Verbosity isn't exactly its problem, its editors are competent, its commandlets are quite capable, it's trivial to author more (and in many ways faster, e.g., adding a cmdlet with arguments is MUCH easier than the handrolled way in bash), you can directly call into a massive stdlib and any code you've written in .NET in any language that is supported by that target (including C#, F#, Nemerle, VB, Clojure and more) with native transitioning for all object types, and a bunch of other good features.
In essence that approach gives the shell a lot more room to do its work without calling out to black-box out-of-conceptual-model plugins, without sacrificing the power of being able to appeal to them when required.
True, but how many of your text-processing utilities need to account for that? Most config file parsers will just continue to work if you insert UTF-8 into the stream, so long as you keep the markers it's working with the same.
>We live in a world where Curling JSON objects or (even more problematically) protobuffs objects is the de-facto way we interact with the vast majority of remote services. Feel free to try and fit every workflow you have into jq. Feel free to keep working in an environment where even functional return values are not allowed and you keep repeating the same commands over and over.
...I don't. Protobufs might have a market share in some places, but JSON has decent shell tooling, and you can often convert JSON to DSV in a pinch. I don't understand what you mean by "functional return values not allowed." Sure, EIAS is in full effect, but Shell pipelines are as functional as they come. And I don't repeat the same commands over and over. If I did, I'd give that set of commands a name.
>Quick question though: how is it this any different from the plugin approach from gulp? :)
Primary difference: The libraries are all already there, you don't have go get apt-get integration for the components you need, and it's often pretty quick to write what you need.
Verobosity is a huge problem. Most of the time with a shell, you'll be writing one-off scripts, or doing something interactive. If you were going to write something more than about 50 lines, or more complex than a couple of pipelines or an execution wrapper, you wouldn't be writing shell. PS is trying to go in a different direction, be a language for even those things that you'd go to another language for, but we've got perfectly adequate languages for that, tons of them, and it fails at things that shell is good at.
I'm not chinese, but I can have sympathy for people who are. Also who are you to tell me [UTF-8 character removed because of bad assumptions in server code] is not a valid bash variable?
> The libraries are all already there, you don't have go get apt-get integration for the components you need, and it's often pretty quick to write what you need.
This is an assumption that's very brittle though. For example, which version of Python did you target with this helper script? Do you know?
And why is that diff for Powershell? It's just a shell with better affordances for automation than bash.
> Verobosity is a huge problem.
If you want a less verbose set of powershell commands, try the latest powershell. Nearly everything you type has a short analogue. Function syntax is shorter. It's easier to do things like automate opening of remote connections (e.g., no need for a shell script to open an ssh connection and toss many streaming outputs into it on an ongoing basis).
> and it fails at things that shell is good at.
You say this but my experience has been the opposite thus far. Making functions actually functions, for example. It's amazing to be able to make a complex jq query repeatable without being forced to pipe everything over an IO stream and collect into a file and then VAR=$(cat result.json).
>This is an assumption that's very brittle though. For example, which version of Python did you target with this helper script? Do you know?
And PS, and every other piece of software out there doesn't have that problem? Don't make me laugh.
>And why is that diff for Powershell? It's just a shell with better affordances for automation than bash.
But shell doesn't nessarily need automation affordances: we have Python/Perl/Ruby for that. What it does need is effective interactive use, which is where I think bash has PS beat.
>I'm not chinese, but I can have sympathy for people who are. Also who are you to tell me [UTF-8 character removed because of bad assumptions in server code] is not a valid bash variable?
That's a problem with Bash itself, which wasn't what I was talking about. My point was that text processing scripts (sed scripts, awk scripts, scripts using cut, etc.) won't choke on unicode, and frequently work pretty much as expected, so long as the tool versions involved can handle unicode, and sometimes even if they can't. If bash has broken unicode support right now, that's not a problem with the paradigm, that's a problem with bash itself, which can be fixed.
And I don't mean to say any one tool is free of version conflicts, but relying on many tools means relying on many versions. Reducing your total dependency graph is a lesson many language environments have taken from the mess of Java and C++.
The instant we start talking about automation then the repeatability of your automation has to be considered.
The shell is arguably an environment for interactive programming as much as it is a shorthand for program invocation.
Here's an earlier comment I wrote about it, which is also in the middle of a discussion about Tup, funnily enough: https://news.ycombinator.com/item?id=4190804
I'm still using Fabricate quite happily today for personal projects in a variety of languages, though I've come to realise that the main downside of it is that programs run slower under strace - a typical gcc invocation to create a single object file might take say, 40% longer. Fabricate.py is also then quite slow to parse the textual output from strace.
Inspired by https://blog.nelhage.com/2010/08/write-yourself-an-strace-in... I wrote a mini-strace that's hard coded to catch the open() and other relevant calls as fast as possible. That helped a fair bit. I've done a lot of hacking on my own copy of Fabricate too which helped a bit, but ultimately I ought to make it do the dependency parsing and saving in a seperate thread, which should help a lot - I'm still not doing parallel builds so I do have spare cores sitting there doing nothing.
I tried a FUSE backend (https://github.com/SimonAlfie/fabricate/pull/60) whereby the filesystem tells you what's being accessed, and that's practically the same as full speed but the downside is that the path jiggery-pokery involved then causes some subsequent problems with eg. gdb not being able to find source files. I've gone back to the strace method for now.
Despite all these niggles, I still use this system wherever possible because it's so, so good having a 'proper' programming language to run my builds, and being able to work forwards in the manner of a shell script. And yeah the automatic detection of changes ultimately saves a lot of time. I can share some of my hacked-up scripts if anyone's interested.
However, using Make + ccache + a custom ar wrapper (port the deterministic flags from Linux to BSD), I got all the speed up of tup/fabricate, using off-the-shelf open source tooling. I've never looked back.
If you have two build targets, do you have two shell scripts? Do you create a library so they share code? That's a dependency graph too.
What's really going on is that the build system doesn't know the true dependency graph until it's run each command once. The first time around, it uses a conservative approximation that you specify.
In tup's case, you don't need to specify any dependencies that aren't themselves generated. That's generally true of header files. If you have a generated header file, you still need to explicitly add the dependency to the tup file [1].
It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic).
The downside is that it's Linux only.
If anyone remembers the name, please do share.
Bear do something similar.
IIRC strace wouldn't work if you used any setuid binaries (horrible but our makefiles used sudo for some things).
There are also some tricky corner cases like sed -i which creates a new file then moves it.
The same might be true for Java, too. Anybody know if modern JVMs bother calling into libc for basic system calls like open, or otherwise obey dynamic interposition of standard library routines?
I haven't tried building it because it seems like it needs FUSE or something similar and FUSE is flaky on OS X. Every time I upgraded OS X I'd have a hard time getting tup to work again.
On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace.
The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These system calls get tracked, and stored in a special build table. One that you don't have to edit.
Then, when you want to run the build again, you just re-run the magic shell. It catches the various commands, and checks their inputs / outputs, and then skips running the command if the targets are up to date.
e.g.
Hmm... "foo.o" is up to date with "foo.c", so I don't need to run the compiler. I just return "success!"That would get rid of all magic build systems. All build syntax. All dependency ordering. The build system would just take care of it itself.
I've played with this before, enough to note that it's likely possible. But I haven't got far enough to publish it.