News for hackers. Stuff that matters.

A comment on Metaflow, Netflix's Python framework for data science, is now open source. Parent comment

Re:Metaflow, Netflix's Python framework for data science, is... by vtuulos on Tuesday December 03, 2019 @07:20PM (#21698372)
Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix.

We could provide similar support e.g. for Airflow, if there's interest.

For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.

[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by thundergolfer on Tuesday December 03, 2019 @08:00PM (#21698625)
> For monitoring, we have relied on notebooks this far.

This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity.

Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?

[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by pela on Tuesday December 03, 2019 @08:12PM (#21698701)
At Netflix, we rely on the workflow scheduler for such alerting and bundle in a layer of triggering mechanism (custom notifications and such)
[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by thundergolfer on Tuesday December 03, 2019 @07:51PM (#21698571)
> Metaflow comes with a built in scheduler.

This isn't a scheduler that will trigger 'flows' though is it? If I search "Scheduler" in your docs the top result is a roadmap item, and searching "Cron" turns up nothing.

Is it true that right now that to run a DAG "every day at 3am UTC" requires an external service?

[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by pela on Tuesday December 03, 2019 @09:28PM (#21699113)
Right now, yes.
[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by thundergolfer on Wednesday December 04, 2019 @12:05AM (#21699745)
Thanks for confirmation.
[ Parent | on HN ]