To give a somewhat longer description of what Pachyderm does/did:
1. Connect data-scale version control with pipeline orchestrator.
2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.
3. Orchestrator only processes changed shards from one run to the next, instead of all of them.
4. Profit.
That's fine for e.g. one mapreduce, but,
1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)
2. If the overall pipeline changes, you can't re-use results from the first change onward
3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.
So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model. And it's so cool that it introduces a pure language at the coordination layer (which this route if architecture basically needs, hence the Nix language/Starlark)...without actually requiring a new language, or even a new file.
It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well.
The main issue so far is that our VM that does the workflow orchestration and the actions runtime are living in separate processes - but we even then, we could do tricks to share memory between them for more efficient / zero-copy data passing... That's not something we have right now though.
With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers, and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^1]). Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".
We never tried to optimize the single-machine case, for better or worse.
[^1]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)
In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.
Where did you hit the most snags with caching?