News for hackers. Stuff that matters.
Re:Metaflow: Build, Manage and Deploy AI/ML Systems by lazarus01 on Thursday July 17, 2025 @06:07AM (#44591595)
I went to the GitHub page. The descriptions of the service seem redundant to what cloud providers offer today. I looked at the documentation and it lacks concrete examples for implementation flows.

Seems like something new to learn, an added layer on top of existing workflows, with no obvious benefit.

Re:Metaflow: Build, Manage and Deploy AI/ML Systems by manojlds on Thursday July 17, 2025 @06:25AM (#44591683)
It's an old project from before the current AI buzz and I rejected this when I looked at it few years back as well with similar reasons.

My opinion about Netflix OSS has been pretty low as well.

Re:Metaflow: Build, Manage and Deploy AI/ML Systems by datadrivenangel on Thursday July 17, 2025 @10:56AM (#44594115)
All the cloud providers have some hosted / custom version of an AI/ML deployment and training system. Good enough to use, janky enough to probably not meet all your needs if you're serious.
Re:Metaflow: Build, Manage and Deploy AI/ML Systems by lazarus01 on Thursday July 17, 2025 @12:57PM (#44595445)
I use google cloud for ML. AWS has a similar offering.

I find google is purpose built for ml and provides tons of resources with excellent documentation.

AWS feels like driving a double decker bus, very big and clunky, compared to google, which is a luxury sedan, that is quite comfortable to take you where you’re going.

Re:Metaflow: Build, Manage and Deploy AI/ML Systems by vibecodemaster on Thursday July 17, 2025 @02:14PM (#44596324)
> redundant to what cloud providers offer today

It may look redundant on the surface, but those cloud services are infrastructure primitives (compute, storage, orchestration). Metaflow sits one layer higher, giving you a data/model centric API that orchestrates and versions the entire workflow (code, data, environment, and lineage) while delegating the low‑level plumbing to whatever cloud provider you choose. That higher‑level abstraction is what lets the same Python flow run untouched on a laptop today and a K8s GPU cluster tomorrow.

> Adds an extra layer to learn

I would argue that it removes layers: you write plain Python functions, tag them as steps, and Metaflow handles scheduling, data movement, retry logic, versioning, and caching. You no longer glue together five different SDKs (batch + orchestration + storage + secrets + lineage).

> lacks concrete examples for implementation flows

there are examples in the tutorials: https://docs.outerbounds.com/intro-tutorial-season-3-overvie...

> with no obvious benefit

There are benefits, but perhaps they're not immediately obvious:

1) Separation of what vs. how: declare the workflow once; toggle @resources(cpu=4,gpu=1) to move from dev to a GPU cluster—no YAML rewrites.

2) Reproducibility & lineage: every run immutably stores code, data hashes, and parameters so you can reproduce any past model or report with flow resume --run-id.

3) Built‑in data artifacts: pass or version GB‑scale objects between steps without manually wiring S3 paths or serialization logic.