Netflix's Metaflow: Reproducible machine learning pipelines

Re:Netflix's Metaflow: Reproducible machine learning pipelines by fwip on Monday December 21, 2020 @05:39PM (#25500534)

One way I deal with this problem in an alternative workflow manager (Nextflow), is by calculating the memory requirement for the ~95th percentile of a job, and submitting with a rule "If this crashes from going OOM, re-submit with memory*N" (up to some max number of retries/RAM). This lets most jobs sail through with a relatively low amount of RAM, and the bigger jobs end up taking a bit more time and resources.

The better your estimator function, of course, the tighter constraints you can use.

[ Parent | on HN ]

Re:Netflix's Metaflow: Reproducible machine learning pipelines by itamarst on Monday December 21, 2020 @08:26PM (#25501840)

In many cases the memory usage is linear with input data, so you can come up with a function that predicts memory usage, add some padding, and then you don't need retries. E.g. this example here: https://pythonspeed.com/articles/estimating-memory-usage/

[ Parent | on HN ]