Re:Metaflow: Build, Manage and Deploy AI/ML Systems
by Bukhmanizer on Thursday July 17, 2025 @06:12PM
(#44598830)
If you’ve tried, has it been clunky to run non-python based workflows? I.e if you want to run bedtools or diamond without having to run a bunch of subprocess.run commands?
[ Parent
| on HN ]
Re:Metaflow: Build, Manage and Deploy AI/ML Systems
by awgl on Thursday July 17, 2025 @06:49PM
(#44599166)
Right, for most of our workflows, we stay in python land, which is great and seamless with Metaflow being in python. But yes, there are occasions that we have to make a system call to run an old R script or even a compiled C++ executable :shrug: (Metaflow does have some native R support tho) I have not had to use the specific tools you called out, bedtools or diamond.
Most of the time this not a blocking problem since each step in a flow is mapped to a Docker image and/or your choice of EC2 instance (e.g. one step on a GPU, another on a memory optimized instance). You can have one step use an image with all of your python-based ML stuff, and another step have a different image with compiled exectuables that are triggered by a system call. If needed, outputs from such a system call would then need to be persisted in a database/S3 or read back into the python flow for persistence. So, it is not as seamless as a flow in all python, but it can work "good enough".