News for hackers. Stuff that matters.

A comment on Netflix's Metaflow: Reproducible machine learning pipelines. Parent comment

Re:Netflix's Metaflow: Reproducible machine learning pipelines by jamesblonde on Monday December 21, 2020 @02:21PM (#25498359)
The Spark community (LinkedIn) developed Dr Elephant to profile jobs and provide suggestions for reducing memory/cpu consumption. Metaflow would need something similar:

https://github.com/linkedin/dr-elephant

[ Parent | on HN ]
Re:Netflix's Metaflow: Reproducible machine learning pipelines by itamarst on Monday December 21, 2020 @03:14PM (#25499030)
I've seen that, yeah. I've already implemented a memory profiler for Python batch jobs (https://pythonspeed.com/fil), but starting to think about how to integrate it into specific pipeline frameworks.
[ Parent | on HN ]