News for hackers. Stuff that matters.
Re:Introducing Drake, a kind of ‘make for data’ by lars512 on Thursday January 24, 2013 @08:30PM (#5113420)
That's exactly what "make -B mytarget" does...

Are you thinking of a particular problematic scenario?

Re:Introducing Drake, a kind of ‘make for data’ by wtracy on Thursday January 24, 2013 @09:11PM (#5113537)
Aboytsov wants to rebuild the target and everything that depends on the target, not rebuild the target and everything that the target depends on. He wants to walk the dependency tree in the opposite direction.
Re:Introducing Drake, a kind of ‘make for data’ by lars512 on Thursday January 24, 2013 @10:46PM (#5113834)
Thanks for pointing that out, much appreciated :)
Re:Introducing Drake, a kind of ‘make for data’ by aboytsov on Thursday January 24, 2013 @08:41PM (#5113452)
No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget and everything that depends on it. Without rebuilding other parts of the workflow that you don't need.
Re:Introducing Drake, a kind of ‘make for data’ by tedunangst on Thursday January 24, 2013 @09:54PM (#5113690)
This is a really weird request. make won't rebuild things that haven't changed, so the default make all rule will only rebuild the things depending on mytarget. Every time you change mytarget, just run make (all) and everything that depends on mytarget (and only those things) will be rebuilt.
Re:Introducing Drake, a kind of ‘make for data’ by aboytsov on Thursday January 24, 2013 @11:12PM (#5113899)
Really?

This is not a weird request, this is one of the most common things we do when we're developing a workflow. You need to do this every time you make changes to code and you want these changes to propagate.

You can't run "make all", because it literally builds everything. You might be working on a specific branch of the workflow, and the overall workflow could be huge. And out-of-date in a lot of places. Or it could contain steps that are very expensive, but not necessary to build for your development purposes (for example, generating a model). This is why exclusions are also important, and make also does not support them.

Make also does not support multiple outputs, and I gave you a prooflink before. And a lot of other things which we think are important, too (I could make a list. I did, actually).

If you like Make, you should continue using it. I think it is a little arrogant on your part to try to explain to us that we simply wasted our time. We built the tool to address the problems we were facing. If you do not face similar problems, by all means, use Make.

Re:Introducing Drake, a kind of ‘make for data’ by tedunangst on Thursday January 24, 2013 @11:46PM (#5114010)
Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.
Re:Introducing Drake, a kind of ‘make for data’ by aboytsov on Friday January 25, 2013 @12:37AM (#5114159)
Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v...

Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process.

You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results?

When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result.

Makes sense?

Re:Introducing Drake, a kind of ‘make for data’ by tedunangst on Friday January 25, 2013 @12:42AM (#5114173)
Thanks.
Re:Introducing Drake, a kind of ‘make for data’ by lars512 on Thursday January 24, 2013 @10:45PM (#5113833)
This is basically the case where you don't (or it's infeasible to) capture the dependences fully, so you want to rebuild everything from target onwards after some change.

In that case: "rm target; make target"

Re:Introducing Drake, a kind of ‘make for data’ by aboytsov on Thursday January 24, 2013 @11:03PM (#5113874)
"make target" will not make stuff down the tree. It will stop at target. Please, try again. :)
Re:Introducing Drake, a kind of ‘make for data’ by lars512 on Thursday January 24, 2013 @11:09PM (#5113888)
Haha, well pointed out. I'm clearly having trouble parsing today.

"rm target; make" can work, but only if you're using a pattern for data pipelines where there is only one default set of downstream targets. If the one Makefile supports a range of downstream targets, then this won't work.

I concede, make doesn't support that operation out of the box :)

Re:Introducing Drake, a kind of ‘make for data’ by aboytsov on Thursday January 24, 2013 @11:14PM (#5113903)
Phew. Thank you.