Data Science Workflow: Overview and Challenges

Re:Data Science Workflow: Overview and Challenges by mbreese on Monday May 15, 2017 @01:27AM (#14339353)

I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for small-medium projects, it's a life saver. Combine that with git, and you've got a pretty robust system. The only real downside is that when you have very large initial inputs, it isn't practical to store those in the same repository.

That is my goto combinations for bioinformatics workflows that don't need to run on a cluster. (A Makefile variant of mine is my go-to for clusters). I've experimented with Docker workflows, which while in theory are nice, you'll always require some sort of organizing script to document the analysis. (Docker is still nice for capturing program versioning/configuration though!)

[ Parent | on HN ]

Re:Data Science Workflow: Overview and Challenges by paulgb on Monday May 15, 2017 @08:26AM (#14341005)

I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/

I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the modified time of the data on the cloud.

[ Parent | on HN ]

Re:Data Science Workflow: Overview and Challenges by in9 on Monday May 15, 2017 @08:52AM (#14341141)

I really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually

[ Parent | on HN ]

Re:Data Science Workflow: Overview and Challenges by paulgb on Monday May 15, 2017 @09:00AM (#14341187)

Yes, I actually only use Python for notebooks, so I mostly ignore the Python specific parts. I think their data/ directory structure (along with keeping it out of the repo) is a sane approach though.

(I don't actually use their template, I just use the directory layout as guidance)

[ Parent | on HN ]

Re:Data Science Workflow: Overview and Challenges by cryptarch on Monday May 15, 2017 @06:02AM (#14340382)

You could use Git-LFS for managing huge data blobs with Git?

[ Parent | on HN ]