Data Science Workflow: Overview and Challenges

Re:Data Science Workflow: Overview and Challenges by paulgb on Monday May 15, 2017 @08:26AM (#14341005)

I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/

I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the modified time of the data on the cloud.

[ Parent | on HN ]

Re:Data Science Workflow: Overview and Challenges by in9 on Monday May 15, 2017 @08:52AM (#14341141)

I really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually

[ Parent | on HN ]

Re:Data Science Workflow: Overview and Challenges by paulgb on Monday May 15, 2017 @09:00AM (#14341187)

Yes, I actually only use Python for notebooks, so I mostly ignore the Python specific parts. I think their data/ directory structure (along with keeping it out of the repo) is a sane approach though.

(I don't actually use their template, I just use the directory layout as guidance)

[ Parent | on HN ]