News for hackers. Stuff that matters.
Re:Metaflow, Netflix's Python framework for data science, is... by zmjjmz on Tuesday December 03, 2019 @11:08PM (#21699492)
Hey I meant to track one of y'all down at the MLOps conference, but didn't get the chance. I've built a very shitty version of a cached-execution DAG thing internally, and one of the design decisions I made was to have it so that parent nodes in the DAG don't need to know anything about child nodes. This allows for larger DAG builders to be more easily subclassed.

MetaFlow doesn't do that -- instead each 'step' has to know what to call next, which means that if I wanted to subclass e.g. the MovieStatsFlow in [here](https://github.com/Netflix/metaflow/blob/master/metaflow/tut...) and say, add some sort of input pre-processing before the compute_statistics call, I'd essentially end up having to either override what compute_statistics does to not match its name _or_ copy-past e that first step just to replace that last line.

I'm sure this design decision was considered and/or that use-case doesn't come up a lot at Netflix (although I've encountered that a lot), or maybe I'm missing something very obvious, but I'd love to hear your thoughts on that.

Re:Metaflow, Netflix's Python framework for data science, is... by savin-goyal on Tuesday December 03, 2019 @11:26PM (#21699574)
We erred on the side of simplicity to keep things manageable for our users.
Re:Metaflow, Netflix's Python framework for data science, is... by beshrkayali on Wednesday December 04, 2019 @12:38AM (#21699887)
That seems more complicated.