Show HN: Orchest – Data Science Pipelines

Re:Show HN: Orchest – Data Science Pipelines by FridgeSeal on Thursday August 13, 2020 @12:13AM (#24139873)

I agree, and so many data science and data engineering tools all seem to revolve around using notebooks, much to my frustration. I’ve worked in places whose data pipelines were built around seemingly infinite notebooks, all containing consistently poor software engineering.

It’s been enough to make me vow to not let people write notebooks that go into prod under my watch lol.

I’m constantly on the watch for software engineering focused tools that solve the issues, rather than data science/engineering focused tools. So many are inextricably linked into python as well, which doesn’t gel nicely with anywhere that has multiple languages in the codebase.

[ Parent | on HN ]

Re:Show HN: Orchest – Data Science Pipelines by calebkaiser on Thursday August 13, 2020 @09:38AM (#24143192)

Shameless plug, but I help maintain Cortex, and "software engineering focused tools that solve (ML) issues" is a neat summary of our entire philosophy. For example, instead of notebooks, our model serving platform (https://github.com/cortexlabs/cortex) uses YAML to structure deployments, and Python scripts to write inference APIs.

It's still inextricably linked to Python, but only for writing your API. It's agnostic as to how the model itself is developed, so long as it can generate predictions.

[ Parent | on HN ]

Re:Show HN: Orchest – Data Science Pipelines by ricklamers on Thursday August 13, 2020 @03:22AM (#24140801)

We'll allow your team to move parts of your code base from notebooks to scripts (.py/.sh/.R) to alleviate your frustration.

That way you can keep using notebooks only for those parts where it makes most sense.

They come together in pipelines that are JSON defined and git-versioned.

[ Parent | on HN ]