Re:Introducing Drake, a kind of ‘make for data’
by xaa on Thursday January 24, 2013 @04:12PM
(#5112250)
As a fellow bioinformatician I can agree that this looks quite useful.
Although (since you mention R), I wonder why there's no love for R in Drake, given that R is perhaps the quintessential data processing language.
Re:Introducing Drake, a kind of ‘make for data’
by dirtyvagabond on Thursday January 24, 2013 @05:09PM
(#5112602)
There is love for R in Drake! As of about an hour ago:
https://github.com/Factual/drake/commit/f63dd2630ca3e5e4a6a6...
I am one of the data engineers at Factual and though I didn't have a role in creating it I definitely enjoy using it on a day to day basis. You begin to see the utility of it when you have a dozen people working up and down a data pipeline and need to coordinate as product specs evolve or schemas change.
I also really like the tagging features - you can add specific tags to different steps in the build and run different "flavors" of your workflow depending upon what is needed. For example, you might build a workflow that collects, cleans, filters, and performs calculations on data from all over the world - but you might also want alternative versions of the build that only work on specific regions or smaller debug datasets. Tags make that really simple to do, even when many steps are shared by the different versions or the dependencies are complicated.