Re:Show HN: Orchest – Data Science Pipelines
by ricklamers on Thursday August 13, 2020 @02:53AM
(#24140650)
Currently, we are not focused on helping DS teams putting models into production.
It's those 'messy' projects with 10K LoC notebook that could easily be broken up into multiple steps (perhaps some of them notebooks, some of them Python scripts with library function usage) that we feel are a great match for pipelines in Orchest today.
When a team is still experimenting with what models to go with (i.e. trying neural networks v.s. decision trees) it can be helpful to have a more structured prototyping environment with reproducability and easier scalability. Which is also where Orchest shines.
If you want to test some of these models in production, you could easily push artificats to endpoints for serving, in the final steps of a pipeline.
It's those 'messy' projects with 10K LoC notebook that could easily be broken up into multiple steps (perhaps some of them notebooks, some of them Python scripts with library function usage) that we feel are a great match for pipelines in Orchest today.
When a team is still experimenting with what models to go with (i.e. trying neural networks v.s. decision trees) it can be helpful to have a more structured prototyping environment with reproducability and easier scalability. Which is also where Orchest shines.
If you want to test some of these models in production, you could easily push artificats to endpoints for serving, in the final steps of a pipeline.